Lessons Learned From the Use of the Most Significant Change Technique for Adaptive Management of Complex Health Interventions

"MSC stories add context to the progress and trends shown by the quantitative data and give good inspiration for local governments and other stakeholders to try the approach [described in the MSC stories] and strengthen their program." - India hub team member..."

Practitioners and researchers are increasingly interested in identifying innovative approaches to monitoring and evaluation (M&E) of complex health interventions. The Most Significant Change (MSC) technique is one such complex-aware M&E tool. It was developed as a means of monitoring changes in a specific development project in the 1990s and introduced to the broader international development sector in 2005. This article shares the experience of and lessons learned by The Challenge Initiative (TCI), which is scaling up evidence-based family planning (FP) and adolescent and youth sexual and reproductive health (AYSRH) interventions in 11 countries in Asia and sub-Saharan Africa. The project has institutionalised the MSC technique as a routine data collection activity to strengthen its monitoring and adaptive management efforts.

Since 2017, TCI has aimed to empower local city governments to rapidly scale up evidence-based FP and AYSRH interventions that meet the needs of their communities. As of this writing, TCI works in 111 local governments, with regional support provided by 5 accelerator hubs: Jhpiego in East Africa (EA), IntraHealth International in Francophone West Africa (FWA), Population Services International in India, Johns Hopkins Center for Communication Programs (CCP) in Nigeria, and Zuellig Family Foundation in the Philippines (joined in 2020 and not included in this analysis since it is new to using MSC). Interested local city governments apply for TCI's support, in the form of financial resources and coaching support, and commit to contributing a portion of the funding needed to adapt, implement, and scale up a diverse range of evidence-based service delivery, demand generation, and advocacy approaches.

TCI has adapted the MSC technique as its primary qualitative data collection approach, complemented by traditional quantitative data. As explained here, "[i]nstead of using precise indicators, MSC uses broad domains of change and asks project stakeholders to share stories of significant change related to those domains. Stakeholder panels then discuss the reported value of the reported changes, select the most significant stories, and provide feedback about their selections and rationale. The stories are intended to be collected at regular intervals to track changes as they are emerging....As such, it helps to support adaptive management. The users of MSC generally report positive experiences with the method, particularly noting unique roles and contributions of the narrative approach. Tonkin et al...noted that the strength of the MSC technique lies in characterizing the change in complex health interventions and confirmed its plasticity, feasibility, and acceptability across different sociopolitical, cultural, and environmental contexts. Consequently, the MSC approach would work well to help evaluate the scale-up of evidence-based interventions across multiple countries....The MSC technique is also easy for participants to understand and could help assess the performance of a program as a whole..."

The qualitative assessment took place in early 2021 to document TCI's use and adaptation of MSC and to determine its added value in adaptive management, routine monitoring, and cross-learning efforts. Focus group discussions and key informant interviews were conducted virtually with staff members, who had collected and selected 260 stories between July 2018 and June 2021 from all the hubs combined.

The article's results section explores the MSC experience, starting from the introduction of the MSC technique to the hub teams and the training process. After securing buy-in, the global-level management team turned the task of identifying and collecting stories over to local hub staff, who provided coaching support to government stakeholders on implementing and scaling TCI's evidence-based interventions. MSC uses an iterative process of selecting and pooling the collected stories to summarise the large volume of stories into a smaller number of widely valued and applicable stories. The hubs then share the selected stories, along with the rationale, with the global-level management team's final selection committee. The hubs organise feedback sessions with their stakeholders in their respective locations, which may highlight, through stories, how each city is doing regarding the programme maturity, for example. The global-level management team and hub teams use additional tools and communication approaches to facilitate further learning and ongoing dialogue about MSC stories.

As several examples from hubs provided in the article illustrate, overall, TCI has had a positive experience with using MSC to facilitate adaptive management in multiple countries. The use of MSC has created learning opportunities that have helped diffuse evidence-based FP and AYSRH interventions both within and across countries. The responsive feedback step in the MSC process was viewed as indispensable to learning and collaboration.

Numerous aspects of the MSC technique make it a useful and valid method for exploring changes in complex environments, which the article outlines. For example, despite northern Nigeria's conservative social norms on family size and FP, TCI has found that stakeholders from the community to the state level have been willing to share their stories because the storytelling and narrative approach is an integral part of their cultural traditions. Notably, MSC stories are shared broadly via multiple channels, including TCI University (the project's online website that synthesises practical FP programme guidance and tools), TCI's community of practice, and social media channels, and they often serve as the needed evidence for other local officials not directly engaged with TCI to adopt and replicate interventions because they too wanted to witness similar results in their context.

There are, however, potential challenges that may arise in using MSC for adaptive management. For example, the data collectors may select to interview only individuals who have had positive experiences. Relatedly, while TCI's use of the MSC technique is intended for the adaptive management purpose, occasionally, positive outcomes from the stories are featured in the project's promotion and communication materials. This unintended consequence may have contributed to the tendency to report stories attractive for communication purposes.

Global health professionals who are embarking on similar initiatives may wish to consider the following enabling factors:

• Ensure leadership buy-in and a shared vision for using MSC for adaptive management: TCI's hub leadership explained to the teams involved in collecting MSC stories that the stories would help them perform their jobs better and improve project outcomes.
• Make data collection part of everyone's job, and strengthen staff capacity to collect stories with support and feedback.
• Develop standard guidance to integrate MSC in routine monitoring, and allow teams to adapt the process.
• Ensure collection and learning from the community level to higher levels of the health system.
• Triangulate data to support significant changes reported in stories, and speak to a range of decision makers.

In conclusion, this assessment suggests that "the MSC technique is an effective qualitative data collection tool to strengthen routine monitoring and adaptive management efforts that allows for flexibility in how project stakeholders implement the process. The MSC technique could be an important tool for global health practitioners, policy makers, and researchers working on complex interventions because they continually need to understand stakeholders' needs and priorities, learn from lessons and evidence-based practices, and be agile about addressing potential challenges."