News

USAID’s Africa Bureau held an Evaluation Summit last week with USAID mission staff and other USAID professionals working to offer new tools and methods to improve evaluations in Africa.  The three-day workshop addressed the themes of how to improve evaluation quality and how to use evaluations to promote organizational learning.

On Thursday, the last day of the Summit, USAID invited The Mitchell Group to participate in the Evaluation Summit Marketplace. At the Marketplace, TMG had the opportunity to nmeet and talk to USAID mission and HQ staff to share our work and disseminate evaluation capabilities, practices, frameworks, and tools that can be used to advance USAID evaluations and provide evidence to improve programs.

TMG chose to showcase an impact evalaution baseline study implemented in the Sahel region of West Africa for the Sahel Resilience and Learning Project (SAREL) in Niger and Burkina Faso.

Figure 1: TMG’s John McCauley and Dalton Brantley showcase TMG’s innovative evaluation methodologies.

The baseline collected data for impact evaluation of a USAID project entitled Resilience in the Sahel Enhanced (RISE). The RISE project comprises Food for Peace activities implemented by Catholic Relief Services, Mercy Corps, Save the Children, and ACDI/VOCA. It also includes the Resilience and Economic Growth in the Sahel – Enhanced Resilience (REGIS-ER) project, whose overall objective is to increase the resilience of chronically vulnerable populations in the agro-pastoral and marginal areas of Burkina Faso and Niger, and the Resilience and Economic Growth in the Sahel – Accelerated Growth (REGIS-AG) project.

The impact evaluation utilized a two-stage baseline sampling procedure that captured responses to (1) a village questionnaire, and (2) a household questionnaire. A total of 100 villages were surveyed with 25 households in each village with a treatment group comprised of high intervention villages (33% of the survey) and a comparison of low intervention villages (67%). In the village questionnaire, a group of 5 to 10 respondents were chosen who have lived in the village for substantial amount of time. Participants included village chiefs, opinion leaders and civil servants among others.

Figure 2: Graphical representation of a longitudinal survey. The blue and red bars show the progress of each village for a given indicator and the boxes represent the groups of participants in each village to survey.

While the comparison between high intervention villages and low intervention villages is standard in impact evaluations, for reasons of attrition, it is frequently not possible to track the same respondents over time.  But with a 2-stage procedure, it will be possible to perform a longitudinal study  at the village level, in order to more rigorously monitor outcomes over the life of the project. At the household level, longitudinal studies can be difficult, expensive and subject to respondent attrition resulting in a 40-50% loss of data. At the village level, as demonstrated in Figure 2, while the individuals (white dots) may change, the unit of analysis becomes the group of people within the village and thus holds increased consistency over time.