"Beyond Gut Feeling" Report on the arepp: Theatre for Life Evaluation Project 2002-2004

About arepp:Theatre for Life

arepp:Theatre for Life is an edutainment non-governmental organisation (NGO) which has been operating nationally in South Africa since 1987. Using live, dramatic presentations, African Repertory Educational Performance Programme (AREPP): Theatre for Life travels to schools providing interactive, social life-skills education to school-going youth with the goal of enabling informed choice and developing resilience within the youth.

Presented in the real-life context of the audience, there are two aspects: the live performance of a play, specifically designed for the targeted age group to foster identification and emotional recognition; followed by a facilitated peer discussion, to encourage debate, and further cognitive personalisation, internalisation and contextualising of the issues presented. These presentations aim to display, encourage, demonstrate and examine life-skills at work, in ‘real life’ situations, and most importantly, according to arepp:Theatre for Life, in context.

Executive Summary

This evaluation project aimed to move beyond “gut feeling” to a more rigorous and comprehensive evaluation of the impact of African Research and Educational Puppetry Programme (AREPP): Theatre for Life in increasing a sense of self-efficacy in the school-going youth with whom it works. It was conducted by Nell and Shapiro, evaluation consultants. Beginning with 7,780 learners (this dropped to 3,902), the same group of learners were monitored over a period of three years (from 2002 - 2004), during their Grade 10, 11 and 12 years. This was done by means of a survey questionnaire, based on indicators of a sense of self-efficacy, administered before the initial AREPP: Theatre for Life intervention, before each of two subsequent interventions, and a few months after the final intervention (there were three interventions in all). Results were cross-tabulated against certain variables in learner and school profiles.

The changes on all the indicators across the total AREPP:Theatre for Life original cohort for the three year period were statistically significant at a 95% confidence level. These results were triangulated against a control group and against learner focus group input and educator interviews. Although it is impossible to isolate the attribution of impact in a complex and interactive social context, the results were encouraging and there was clear evidence that the AREPP:Theatre for Life interventions do increase a sense of self-efficacy. In addition, the cross-tabulations and qualitative input provided invaluable information about context and learner differences that will enable AREPP:Theatre for Life to improve its presentations and the way in which it embeds its work at the local level.

While it may never be possible for organisations working in the social sphere and dealing with the realities of human interaction and a multitude of interventions, to say conclusively: “if not x then not y” where x is the intervention and y is the impact. The study was hoping to show from analysis of the data that “if x then y”, where x was the AREPP:Theatre for Life intervention and y was the desired impact of ‘an increased sense of self-efficacy’.

In addition, we wanted to be able to say “if x+a+b+c then better y”, where a, b and c were different variables or conditions in which x was offered. This would firstly enable us to confirm that AREPP: Theatre for Life was not wasting its time nor the donors’ money and, secondly, it would enable AREPP: Theatre for Life to adjust the product to take advantage of different facilitating circumstances, or to address the disadvantages of other hindering circumstances.

The learnings from this analysis are not only about change and impact, they are also about where AREPP: Theatre for Life has most success, where the presentations are not necessarily having the desired impact and in what sort of context the organisation is most likely to achieve its goals. This makes the usefulness of the whole process far greater than it would have been had it shown only that there was a significant change on all the indicators (which it does, in fact, do).

For the purposes of the study, five statements for each of the five AREPP: Theatre for Life indicators of impact were created on which learners could rate the extent of their agreement or disagreement. It is worth noting that one of the most exciting spin-offs of the evaluation project has been the further and clearer refinement of what AREPP: Theatre for Life defines as impact. This has led to a necessary clarification of the indicators and a review of appropriate statements which have already informed the work of AREPP: Theatre for Life and will certainly inform further efforts in evaluation.

Questions about the reliability of the statements grouped under each indicator to measure that indicator were raised by the reliability analysis done on the responses to the statements. Here we are concerned with whether the statements grouped under each indicator as a measurement of the indicator, do, in fact, measure it. The analysis showed that there was adequate internal consistency in the responses on the indicators of “control” and “self-esteem/self-awareness”. It showed that there was some consistency on “constructive relationships” and “options and consequences”, but not sufficient to warrant confidence in terms of reliability that all the questions related to these indicators. On the indicator of “tolerance” the correlation between the responses to the five statements was weak. What this means is that the statements grouped under the indicators did not necessarily all measure the same indicator (tolerance), although the statements themselves may have remained (and largely did remain) reliable indicators of “things AREPP: Theatre for Life wanted to measure”.

Overall, we think that the plays presented by AREPP: Theatre for Life deal with the issues with which AREPP: Theatre for Life is concerned, and that the statements remain valid indicators of what it is trying to achieve, and what it was trying to measure, even when they are not necessarily grouped correctly. We take this grouping issue into account in interpreting the analysis of the data.

Click here to download the full Executive Summary as a PDF file from the AREPP website.

Click here to download the full report as a PDF file from the AREPP website.