Evaluating the Impacts of Media Assistance: Problems and Principles

"While some form of evaluation has always been a requirement of development projects, in the media assistance field this has predominantly been limited to very basic modes of counting outputs, such as the number of journalists trained or the number of articles produced on a topic. Few media assistance evaluations manage to provide sound evidence of impacts on governance and social change."

This paper attempts to contribute to rectifying the problem of inadequate media assistance impact evaluation by going beyond collating evaluation methodologies and methods into principles, toolkits, and guides, instead suggesting that specific types of contextual factors must be known and understood. "The critical analysis of media assistance evaluation in this paper uncovers the ways that donor interests and bureaucratic systems and shape evaluation practices. It also points to how the continuities in the positioning of media assistance goals and objectives influences evaluation design."

Author Jessica Noske-Turner introduces the paper by explaining that "common strategies in media assistance include: building the capacity of journalists and other media professionals, strengthening professional associations or educational institutions, advocating for policy and legislation reforms, promoting citizen voice through the media, and providing infrastructure or equipment supports." Some of the largest media assistance organisations she cites are: Internews, BBC Media Action, Article 19, IREX, and Panos.

The findings in this paper are drawn from a research project that used a multi-phased, qualitative research methodology. Noske-Turner began with a document analysis of 47 evaluation reports sourced from the Catholic Media Council (CAMECO) resource library (30), the Communication Initiative (13), and manual searches (4). This was followed by semi-structured interviews with 10 evaluators. Finally, she undertook a case study of the Cambodian Communication Assistance Project (CCAP); this involved document analysis, participant observation, and semi-structured interviews with project stakeholders and observers (48) and other media and communication for development (C4D) actors in Cambodia (5).

To provide a frame of reference to build her argument, Noske-Turner outlines some of the academic and industry responses to the problem of media assistance evaluation, such as: the "Caux Principles", which emerged from a 2010 meeting of an international and multi-agency group of practitioners and scholars; the CAMECO's mediaME wiki, which was intended as an open space for practitioners to share their resources and expertise on methods and approaches; and the Global Forum for Media Development's (GFMD) toolkit, which focuses on comparing international indicators relevant to media assistance for the purpose of a needs assessment.

Though acknowledging that some of these outputs (several more are outlined in the paper) have been useful, Noske-Turner argues that inadequate media assistance evaluation is not simply a problem of evaluation methodology. She outlines 3 "underlying tensions and challenges that stifle implementation of effective practices in media assistance evaluation." They are as follows:

1. The problem of conceptual ambiguity: Citing research carried out by Jones and Waisbord that found that "'indicators' in media assistance projects tend to be measured in terms of quantified outputs, such as the number of journalists trained, rather than any real attempt to address the stated goals", Noske-Turner looks to Manyozo's rubric as a foundation for her case study. She explains that Manyozo's approaches involve: (i) media for development, where the focus is on content in order to educate and inform; (ii) media development, where the focus is on industry with the goal to improve governance; and (iii) community engagement, where the focus is on dialogue with the aim to foster participatory decision making and self-determination. Noske-Turner uses this framework to analyse Cambodia's CCAP project, which aimed to introduce talkback radio formats to government-owned provincial radio stations. The CCAP project design document described objectives that approximated each of the 3 approaches that Manyozo outlined. "Applying this structure to an analysis of CCAP's evaluation activities illuminated the interplay of theories, but also the confusion and complexity that can occur when seeking to create indicators and assign methods without theoretical clarity....Here then, the root of the problem lay in a difficulty in operationalising the key term, 'governance', and translating the indicators provided into methods and evaluation questions....For a project like CCAP, where the aim is to introduce talkback or other discussion program formats, I propose that increased attention to the processes of two-way communication (between citizens and authorities), understood in terms of Manyozo's community engagement approach, would significantly enhance the design of evaluations. In this way, dialogue and participation would be understood as contributing to good governance processes. Pivoting the primary model of change in this way has several implications that are important to consider. First, it suggests very different types of indicators and evaluation questions, such as attention to changes in both the governed and the governors. Second, it redefines impacts in terms of changes in social processes, including changes in power distributions, voice, dialogue and relationships, which each contribute to the good governance processes such as accountability and transparency. This emphasis on the communication process itself is a significant shift in the ways that the development industry is used to treating the idea of impacts..."
2. The influence of the procedures associated with evaluation of media assistance - "bureaucracies' demand for a document" - on the resulting evaluation practice: (i) The implementation of evaluation procedures is often triggered by systematised time periods in the project cycle, rather than by active decisions about when an evaluation is needed, which means that this trigger often comes too soon for any results to be measurable; (ii) "standardised procedures are increasingly used as an alternative to spending time working through the specifics of evaluation."; (iii) "Choices are constrained by the allocation of resources for evaluation at the end of the project."; (iv) The use of commissioned outsiders, also possibly viewed as an evaluation "procedure". This is, according to Noske-Turner, potentially fraught with challenges such as the perception of "ownership over commissioned evaluation reports, which challenges the consultants' roles as independent evaluators."
3. The complex epistemological and political undercurrents of media assistance evaluation: Noske-Turner argues that a focus on toolkits only can lead to methods being used without engagement with broader evaluation theory and scholarship. "There are three primary epistemological tensions in evaluation that deserve more attention in the debates relating to media assistance evaluation. These relate to the setting evaluation objectives, the framing of change processes, and the use of participatory and collaborative evaluation approaches." With regard to the latter of the 3 tensions she cites and returning to the CCAP case study, she writes: "Close analysis of the CCAP project, uncovered the existence of participatory principles and processes, even though the formal language of participation was not used to identify it as such. Participation in evaluation was repeatedly shown to improve the engagement, the sense of ownership, the quality of insights and the relevance of the strategies identified for solving problems or weaknesses....Crucially, however, the research raised some challenges in incorporating participation including the potential to over-burden participants and, more importantly, the potential risks of exposing participants to uncomfortable, even unsafe, situations, given the highly politicised environment (common to many media assistance project contexts). Therefore, participation can be useful, but not everything must be participatory if it is not feasible or practical."

The paper concludes by outlining 4 principles for media assistance evaluation: (i) "Plan early, evaluate regularly, adapt periodically"; (ii) "Involve stakeholders in planning and evaluation"; (iii) Ensure that evaluation designs are theoretically informed and framed; (iv) Consider consultants as facilitators and coaches during the evaluation design process, and later as scrutineers over internal research and evaluation.