Changes in On-time Vaccination Following the Introduction of an Electronic Immunization Registry, Tanzania 2016-2018: Interrupted Time-series Analysis

"EIRs [electronic immunisation registries] have the potential to improve vaccination timeliness but need to be used consistently to provide accurate metrics on target populations."

The use of digital health interventions (DHIs) among immunisation programmes has become more widespread and shown promise for improving vaccination uptake and adherence to immunisation schedules. The objective of this study was to assess the impact of a DHI on immunisation timeliness by investigating the proportion of on-time vaccinations following introduction of an electronic immunisation registry (EIR) in Tanzania.

Beginning in 2013, the Better Immunization Data (BID) Initiative partnered with the Ministry of Health (MOH) in Tanzania to address key challenges in immunisation data collection, quality, and use. By 2018, the intervention package to address the challenges identified was deployed in the regions of Arusha, Dodoma, Kilimanjaro, and Tanga. The EIR allowed health care workers (HCWs) to register children, record vaccinations administered, identify vaccinations due, and generate aggregate facility-level reports that fed into the health management information system (HMIS). The intention of the EIR was to replicate and eventually replace the use of paper-based data collection tools used in immunisation clinics, and in March 2018 facilities in the Tanga region started transitioning to entirely digital reporting. In addition, beginning in April 2018 SMS (text messaging) vaccination reminders were sent automatically through the EIR's server to the caregivers of children with delayed vaccination visits.

The study's theory of change held that, by introducing the BID intervention package, HCWs should have improved access to and use of data to identify vaccines due and follow-up with defaulters; therefore, vaccination timeliness should improve. This theory makes the following assumptions: a) every child seen at a facility is entered into the EIR; b) all required information is entered into the system; and c) HCWs actively complete follow-up with defaulters, including through automated SMS reminders.

The researchers used secondary data from the EIR as implemented in 1,006 health facilities (251,815 children with vaccinations recorded in the EIR) to test the following hypothesis: the introduction of the EIR in Tanzania led to improvement in vaccination timeliness after 3,6, and over 6 months post-introduction. They used an interrupted time-series (ITS) analysis study design to measure changes over time. They set up their primary analysis based on their hypothesised theory of change but quickly found substantial levels of missingness in the data either due to a child not receiving a vaccination or lack of recording. As such, they conducted a sensitivity analysis to attempt to address this limitation using only registered children with a subsequent vaccination dose recorded in the EIR.

The primary analysis revealed a decrease in the proportion of on-time vaccinations following EIR introduction. However, the researchers urge caution in interpreting these findings, as additional information is needed to understand if the changes observed reflect true estimates of timeliness or if they reflect incompleteness in EIR vaccination records and biases from the data capture process. Among the implementation challenges that may have affected the completeness and/or accuracy of the data: inconsistent use of the EIR over time; the official requirement of completing dual data entry with the paper record remaining the official record, potentially causing HCWs to ensure paper records were more complete than EIR records; inconsistent use of unique patient identifiers causing individuals to have multiple IDs; and/or poor data entry practices, leading to inaccuracies due to workflow or training issues.

In contrast, the sensitivity analysis indicated improvements in timeliness among those children with complete vaccination records. For example, the researchers estimated significant differences in the likelihood of on-time diphtheria, tetanus, and pertussis (DTP1) vaccination post-EIR introduction compared to pre EIR, with a 19% increased likelihood of being on time 4-6 months post EIR (odds ratio (OR): 1.19, 95% confidence interval (CI): 1.11-1.27) and 34% increase more than 6 months post EIR (OR: 1.34, 95% CI: 1.23-1.45). Significant increases in the likelihood of on-time vaccinations were estimated for 0-3 months and over 6 months following EIR introduction, with 2% increase per month (OR: 1.02, 95% CI: 1.00-1.04) and 6% increase per month (OR: 1.06, 95% CI: 1.05-1.07), respectively.

In reflecting on the findings, the researchers note, in part: "Simply having HCWs utilize an electronic tool will not on its own increase timeliness of vaccinations;...HCWs would have to use the information in the EIR to encourage caregivers to bring children on-time for their next scheduled immunization appointment and follow-up with defaulters." They also suggest that: "Digital systems can provide a secure location for record storage, increase patient trust in the healthcare system, improve data quality and accessibility, and can reduce the burden of data management activities, freeing up time for staff to focus on patient care....As the health benefits of DHI may take 3-13 years to be observed, the importance of these more proximal process outcomes should be acknowledged and these metrics used in DHI evaluations..."

The researchers conclude that decreases in vaccination timeliness following EIR introduction were "likely due to inconsistent data entry and use of the EIR, rather than a true decrease in the population. To interpret...[these] findings more accurately, contextual information about EIR implementation would have helped to provide a comprehensive understanding of the validity of this finding....Use of individual-level RHIS data generated from these systems can provide greater insight into immunization program performance and ultimately help reduce gaps in vaccination coverage once implementation challenges are overcome."