Assessing the Effectiveness of a Web-Based Vaccine Information Management System on Immunization-Related Data Functions

In Tanzania, the United States Agency for International Development's (USAID) Maternal and Child Survival Program (MCSP) provides technical support to the Ministry of Health, Community Development, Gender, Elderly, and Children (MOHCDGEC) in 7 regions. After one year of piloting, the Immunization and Vaccine Development (IVD) Program of the MOHCDGEC commissioned this implementation research study to examine the effectiveness of a vaccine information management system (VIMS) in improving immunisation programme data quality. The specific study objectives are to: determine changes in data quality caused by VIMS use by comparing data quality indicators between VIMS pilot and non-pilot districts; explore the usability of the VIMS system by asking VIMS users to describe their experience using it; and examine the information management experiences of staff in non-VIMS districts.

In the past, IVD primarily collected programme-related data through the use of 3 tools: a district vaccine data management tool (DVDMT), a stock management tool (SMT), and a cold chain equipment inventory management tool (CCEIT). However, assessments indicated that the quality, consistency, accessibility, and timely availability of data from these tools could be improved. Therefore, the IVD piloted a new VIMS that combines the 3 tools and streamlines data collection into one visualisation platform to reduce the duplication of efforts for data management required by health workers. VIMS was developed under IVD leadership by John Snow, Inc. (JSI)/DELIVER and in partnership with the Clinton Health Access Initiative, PATH, World Health Organization (WHO), and MCSP. The new data management system provides IVD programme stakeholders with a web-based "one -stop" source of information on vaccine and immunisation commodities, cold chain assets, and routine immunisation data. Beginning in August 2016, IVD piloted VIMS in 44 councils/districts in 7 Tanzania mainland regions.

Conducted in July 2017, this cross-sectional, post-test study featured a non-randomised intervention/control design. The intervention arm included 4 of the 7 VIMS pilot regions from Tanzania mainland: Arusha, Mtwara, Mwanza, and Njombe. The control arm included 4 non-VIMS pilot regions: Mbeya, Dodoma, Shinyanga, and Tanga. The goal was to examine the effectiveness of VIMS in improving immunisation programme data quality and usefulness of IVD programme data relative to the DVDMT, SMT, and CCEIT system by comparing VIMS pilot and non-pilot districts.

Of the 19 respondents from VIMS pilot regions, 17 respondents (90%) received the initial VIMS training, and all respondents received some combination of initial, refresher, and/or on-the-job training. The majority of respondents (63%) said they were satisfied with the quality of training received, but the same number said they still needed additional training. When asked what topics they wanted additional training on, respondents mentioned stock management, generating reports, voucher preparation, cold chain management, and VIMS registration. In terms of ease of learning VIMS, 79% of respondents said they learned to use VIMS quickly, 90% said that VIMS manuals are clear, 84% said they could easily remember how to use VIMS, and 74% said they had adequate support in using VIMS.

Most VIMS users reported feeling positively toward the new system, saying that it is easy to use and streamlines multiple data collection and reporting functions, easing the burden on health workers for data collection. Respondents liked that VIMS: allows auto-calculations and can generate real-time reports; prevents submission of incomplete reports; makes it easy to detect reporting errors; is accessible anywhere that you have an internet connection; has the potential to make stock status management easier; and provides dashboard displays of statistics and trends that are helpful in decision -making.

Regarding VIMS usability, perceptions were generally positive, but there were concerns about VIMS being "unnecessarily complex" (32%), "cumbersome relative to the old system" (37%), and having "confusing" error messages (26%). Nearly two-thirds of respondents (63%) said they had difficulties using VIMS because of poor internet connections and perhaps related, 16% said that VIMS's speed was "too slow". For example, some respondents mentioned that, due to internet connectivity challenges, it sometimes took a long time for VIMS to become updated with newly entered data. Despite these concerns, 95% of respondents reported feeling "very confident" using VIMS, 89% said that VIMS was "user-friendly", 95% said they would "like to use VIMS frequently", and 95% said they were "satisfied" with VIMS.

The study found that VIMS did not have any statistically significant effect on improving IVD programme data reporting accuracy, consistency, or timeliness relative to the current DVDMT/SMT/CCEIT-based system. Nor was VIMS found to significantly improve district performance in vaccine stock management compared to use of the DVDMT/SMT/CCEIT tools. Several operational, infrastructural, and/or study-related factors, examined in the report, help explain why this was the case.

Beyond easing the data collection burden, for Tanzania to see additional benefits of using VIMS that other countries, such as Ethiopia and Paskistan, have observed implementing similar electronic vaccine logistics management information systems (LMISs), further adjustments and investments in VIMS refinement and training/mentoring support for VIMS users are needed. To maximise the benefit of future investments, the following recommendations are offered:

• Refine the VIMS tool as needed to address technical issues identified by users in this study (e.g., simplifying certain processes, functions, or displays and making error messages more clear and user-friendly).
• Provide follow-up training and frequent on-the-job training to VIMS users (including health facility workers, the sources of data entered into VIMS) at all levels and focusing on competency-based training to build skills. In addition, prioritise inclusion of VIMS-related items in supportive supervision visits (and in supportive supervision checklists) at sub-national levels. This is in recognition of the fact that effective use of VIMS (or any information technology solution) is not solely a "tools" or "technology" issue but also a "systems" issue, where stakeholders must be continuously supported in their technology use.
• Increase the provision of feedback on IVD programme performance from the national to sub-national levels (and vice versa), so that users can be engaged in a discussion about data, data use in decision-making, and continuous performance monitoring and improvement. As noted in the comprehensive multi-year plan (cMYP), activities like annual IVD evaluation meetings have not taken place. If resumed, review meetings would facilitate stakeholders at all levels in discussing VIMS (and other) data.
• Update VIMS as needed to accommodate new vaccines. According to the 2016 -2020 cMYP, Tanzania will introduce 7 new vaccines (inactivated polio, bivalent oral polio, hepatitis B birth dose, yellow fever, human papillomavirus, meningitis A, malaria) by December 2020. Continuous updating of VIMS to include new vaccines will ensure that it stays relevant and useful to IVD stakeholders.
• Advocate for/mobilise additional resources to further integrate VIMS with the health management information system (HMIS)/ District Health Information Software 2 (DHIS2), such that these data management systems are streamlined and linked.
• Integrate VIMS with the Electronic Immunization Registry (EIR) in collaboration with the MOHCDGEC and PATH-led Better Immunization Data Initiative (BID). In Arusha and Tanga regions, health facility staff at the service delivery level currently use the EIR to digitise IVD programme data (going from a paper-based to an electronic system).
• Conduct further VIMS-related implementation research studies to assess whether using VIMS saves time relative to using the DVDMT, SMT, and CCEIT tools and to determine the impact VIMS usage has on stakeholders' data use and decision-making. Study findings could then be used to further strengthen the VIMS system.
• Continue to document VIMS scale-up to inform future Tanzanian implementation as well as potential VIMS adoption and adaptation in other countries.