Estimating Causal Effects from Family Planning Health Communication Campaigns Using Panel Data: The "Your Health, Your Wealth" Campaign in Egypt

"The difficulties of evaluating family planning communication programs may be surmountable using panel data and analytic methods that address both observed and unobserved heterogeneity in exposure."

This paper assesses the effects of exposure to a family planning health communication programme - the "Your Health, Your Wealth" (Sahatek Sarwetek) national multimedia campaign in Egypt - on a set of family planning outcomes, including current use of modern contraception, measures of interpersonal communication regarding family planning, and attitudes towards birth spacing. The aim of the paper is in large part methodological: to control appropriately for non-random (self-reported) exposure to the programme in order to obtain more accurate measures of the programme's effects.

The introductory section shares what is at the heart of the problem for the evaluation of many large-scale health communication interventions: the inability or impracticality of using experimental designs in which individuals are randomised into exposed treatment groups and unexposed control groups. This is due to the fact that: the interventions often cover entire countries, potentially exposing all "targeted" individuals; localised interventions risk contamination across geographic areas; or ethical concerns proscribe limiting dissemination of health messages to a subset of potential beneficiaries. In the absence of randomised control designs, evaluations of health communication programmes frequently adopt alternative methods to generate inferences surrounding causal relationships - several of which are discussed here.

The approach used in the present study is to make use of panel data to estimate the effects of "Your Health, Your Wealth" (Sahatek, Sarwetek) while controlling for unobserved heterogeneity between exposed and unexposed respondents, thereby potentially improving upon methods that make no such controls or that rely solely on cross-sectional data. Specifically, the data come from 2 waves of the Menya Village Health Surveys conducted in 7 villages of Menya Governorate in Egypt in 2004 and 2005; the focus is on ever-married women aged 15-49 years. Several estimation methods are used to attempt to develop and to compare estimates of the causal effect of exposure to the national multimedia health communication campaign: (1) a single-equation cross sectional estimator (using endline data), (2) matching on the propensity score of exposure, (3) a difference-in-differences (DID) estimator, (4) a fixed effects estimator, (5) a fixed effects, instrumental variables estimator, and (6) a bivariate probit model that models simultaneously both the outcome and exposure equations and allows for correlation in unobservables across the 2 equations.

A component of the Communication for Healthy Living (CHL) project in Egypt (see Related Summaries, below), the campaign grew out of a cooperation of the Egyptian Ministry of Health and the United States Agency for International Development (USAID). CHL in turn was one part of the Health Communication Partnership (HCP), a global health communication initiative funded by USAID. "Your Health, Your Wealth" involved national multimedia and community-based interventions aimed at encouraging families to engage in healthy behaviours at different points in the life stage. A key component of the overall communication strategy was the "Mabrouk!" (Congratulations!) Initiative, geared towards newlyweds as a strategic entry point for encouraging behaviours that promote healthy families, including events such as pregnancy, labour and delivery, postpartum infant care, family planning, and overall family health.

All villages were exposed to the national campaign, which was broadcast on national television. However, 5 of the 7 villages received more intensive community-based interventions from CHL, while 2 villages were used as comparison villages. The intensive community-based interventions were implemented through community development associations (CDAs), and included activities such as newlywed visits, pregnancy classes, safe delivery referrals, and postpartum home visits. The home visits provided an opportunity to reinforce national media messages through interpersonal communication, and to encourage women to start using family planning within 40 days after the delivery and to space their children 3 years apart.

As the measure of exposure to "Your Health, Your Wealth", the researchers use the variable from the 2005 wave of the survey indicating whether or not the respondent reported having seen campaign messages in the last 12 months and specifically mentioned that those messages pertained to either "birth spacing" or "family planning use in the 40 days following birth". In the sample of 2,088 women in 2005, 378 (18.1%) recalled having seen either messages; 321 (15.4%) recalled the messages related to postpartum family planning use, and 151 (7.2%) recalled the messages related to birth spacing.

At the endline, statistically significant differences were observed for 3 of the 4 outcomes. For instance, DID estimators find that exposure to the campaign increases the likelihood of spousal discussions by 14.4 percentage points (pp.) (standard error (SE) = .039, p<0.001) but has no effect on contraceptive use. In contrast, the fixed effects, instrumental variables estimator, controlling for unobserved heterogeneity, finds a large, statistically significant effect on modern contraceptive use (27.4 pp., SE = 0.135, p = 0.043).

The results, in short, provide evidence that "Your Health, Your Wealth" succeeded during the short time period of this study to achieve change in family planning outcomes. The models that made use of the panel nature of the data set - DID, propensity score matching (PSM), and fixed effects logit - provided similar results in terms of direction and levels of statistical significance but the magnitudes of effects often diverged widely. However, the researchers find that "estimates of program effects based on the perhaps naïve assumption that exposure is largely random once a limited set of observed covariates are controlled for may substantially underestimate program effects." In fact, they find that the magnitude of the underestimates could be on the order of 3- to 5-fold. For example, for 2 common methods assuming exogenous exposure - DID models and PSM - exposure to the campaign was associated with an increase in modern contraceptive use of 2.9 and 4.0 percentage points respectively. In contrast, for the methods controlling for endogenous exposure - the bivariate probit and fixed effects IV estimator - the effect sizes were 23.1 and 27.4 percentage points respectively.

In conclusion: "The difficulties of evaluating family planning communication programs may be surmountable using panel data and analytic methods that address both observed and unobserved heterogeneity in exposure. Not controlling for such effects may lead to substantial underestimates of the effectiveness of such campaigns."