Can Machine Learning Help Us Measure the Trustworthiness of News?

"[I]mpartial, fact-based news is a powerful indicator for the quality of media and the vibrancy of an information ecosystem."

In Mozambique, IREX's Media Strengthening Program (MSP, funded by the United States Agency for International Development, USAID) supports Mozambican professional and community journalists and their media platforms to provide high-quality information to citizens. As part of this project, IREX worked with Lore to test whether machine learning can help detect news articles that contain journalists' opinions and biases. This case study describes the process of undertaking the experiment (a minimum viable prototype, or MVP), along with findings and lessons learned.

IREX explains that, despite its importance, measuring the quality of media content using traditional methods can be resource-intensive and time-consuming. IREX's Media Content Analysis Tool (MCAT) is a framework that applies content analysis to systematically score selected text against 18 indicators of media quality. Scores are calculated for each story and averaged across a representative sample of stories over a period of time. The framework has been tested and used for decades in multiple countries. When evaluators are sufficiently trained, the MCAT yields reliable insight into the trustworthiness of news, but it can be a slow, inefficient, inconsistent, and expensive process.

Over a period of 6 months, the MCAT evaluators trained Lore's machine learning software, called Salient, to detect opinions in the text of news articles. This test - whether the reporter inserts his or her own opinion into a news article - is one of the 18 indicators that the MCAT measures. Identifying an opinion in a news article involves reading the article and looking for subjective language or first-person perspective; often these phrases are associated with opinions that do not belong in fact-based news reporting. The team used software to scan websites for 9 print media outlets in Mozambique and imported more than 1,200 articles into machine learning software. Using the software's "highlighter" tool, evaluators clicked sentences in the articles to show the software examples of opinions. Using these examples as a guide, the software identified patterns and searched for other sentences that were similar. Evaluators reviewed sentences that the software flagged as possible opinions and determined whether the suggestion was correct or incorrect. The team conducted this feedback loop 51 times.

The results, which are shared in detail in the case study, suggest that it is feasible for machine learning to help find opinions in news articles. After 16 rounds of "training" the software, the software identified articles that contained opinions with 95% accuracy (and accuracy increased over time). IREX concludes that the software identified these articles reliably enough to apply it to monitoring and evaluating media content, at a scale and speed that outpace conventional human evaluators.

Based on these results, IREX feels that there are promising implications for media support programmes. For example, "machine learning could help us explore whether there is a relatively higher ratio of opinionated content in news articles for certain topics in a country or region. Answers to questions like this can transform how media practitioners understand and support robust media ecosystems." To cite another example from the media support sector, a non-governmental organisation (NGO) "could have a real-time window of the quality of the hundreds of news articles that its trained journalists were publishing every week. This could help the NGO measure anything from the effectiveness of its training efforts in a particular program to the health and vibrancy of an information system."

There are, however, limitations of the MVP, which are outlined in the case study. For example, other MCAT indicators, like whether the article cites a variety of reliable sources, are not as easy to automate. The software can't yet distinguish whether opinions belong to the author and, further, whether or not an article includes an opinion is a relatively minor indicator of impartiality. Also, this experiment did not eliminate bias; through training the software to look for opinions, the MCAT team actually encoded its existing bias into the software.

IREX contends that the experiment offers several lessons for other NGOs, media organisations, global development practitioners, and monitoring, evaluation, and learning (MEL) specialists who are interested in using machine learning to amplify their work. Some of what they found: "Carefully defining a problem statement, starting small, investing in training on skills to use a new tool, understanding the limitations and advantages of the technology, and communicating effectively and realistically about its potential impact were each an invaluable asset to the successful implementation of this machine learning program." IREX also points to the importance of a strong and trusting working relationship with the partner - in this case, Lore: "Find a technology partner who is genuinely willing and able to interact with subject matter experts (such as a media NGO), provide the necessary training, and navigate some of the technical complexities, pivots, and uncertainties that come with an MVP."

In conclusion, IREX hopes that "this experiment contributes to a growing global commons of shared experiences and good practices among development actors about promises and pitfalls of working with machine learning tools in practice."