Piloting Evaluation Tools for Data Collection and UNICEF’s Role in Providing Quality Assurance

By Cody Minnich, Evaluation Intern, UNICEF Cambodia

On a rainy Monday morning in the Prasat Bakong district outside of Siem Reap, my colleague, Phaloeuk Kong, and I sat in on a focus group discussion(FGD)with a commune council regarding a cash transfer pilot project that had been implemented in their commune. The cash transfer project was designed by the Council for Agricultural and Rural Development(CARD)with support from UNICEF to target pregnant women and children under-five with the intention that the additional income will allow the beneficiaries to improve in the areas of maternal health and nutrition. The primary purpose of this evaluation is to foster learning and improvement within the cash transfer pilot project, and ultimately inform the design of the national cash transfer programme for a nationwide roll-out by the Royal Government of Cambodia. On this morning inside the Commune Development Resource Center, two members of the evaluation team worked their way through the FGD guide with the commune council members.

Phaloeuk and I observed and took notes as Kriti Gupta and Chey Tech, the evaluation team members, conducted the focus group. Kriti moderated the discussion while Tech translated. Although the FGD guide provided direction for the discussion, Kriti asked additional questions, consolidated questions, and rearranged questions where necessary, so the discussion could flow more organically.

Collecting data and generating good evidence is paramount to any evaluation, so the tools to collect data must be both efficient and effective. According to Kriti: “Piloting of data collection tools with respondents who are not a part of your final sample is a very important step towards finalization of the tools. It allows the evaluators to ensure that there are no redundancies in the questions and also that the language and flow of questions is appropriate. To generate good evidence, it is necessary that the data collection tools do not take too much time to be completed; are easy to understand, clear and have a single meaning; and are culturally sensitive and relevant.”

Later in the day, I observed Kriti and Tech conduct a key informant interview(KII) with a health center staff member, while Phaloeuk observed the recently trained enumerators survey the households of the beneficiaries. At the health center, I watched as Kriti and Tech once again utilized their previously developed tool as a guide while adapting the questions as the interview developed to generate the most useful and meaningful responses.

According to Kriti, piloting the various tools confronts some of the major challenges in the data collection process: “One key challenge of data collection, which the pilot helps overcome, is to pre-empt any difficulties which the respondents might face in understanding and/or answering questions. This helps the evaluators to identify the best way to derive certain information. Piloting of data collection tools also helps understand the specific cultural aspects of the particular geographic area where the tools will be administered to allow for sensitivity and relevance.”

At the end of the day, the enumerators, the evaluation team, Phaloeuk, and I reconvened at the UNICEF office in Siem Reap to discuss our observations from the day of piloting tools. Kriti guided the discussion by first asking whether the tools contained any questions that the respondents were incapable of answering. From there, we went through every data collection tool, page by page, ensuring there were not any questions that were confusing, inappropriate, or superfluous. After nearly three hours, everyone had provided their input into the various data collection tools and were satisfied with the refined products.

Phaloeuk and I met with Kriti and Tech again the next morning to provide them with our feedback and observations from the previous day of piloting. Our feedback consisted both of ethical and practical considerations of both the tools and the methodology in using the tools. UNICEF’s role is to provide quality assurance throughout the process and ensure that the final products will produce the most meaningful information possible while following ethical guidelines. On UNICEF’s role in the piloting process, Kriti stated: “UNICEF’s approach to an evaluation is unique and plays a very important role in assuring quality. Constant feedback from the evaluation management team and an expert eye of key stakeholders through the reference group proved to be very helpful in assuring high quality of data collection tools.”

The evaluation team utilized the finalized versions of the data collection tools and successfully completed the field work phase of the evaluation.

The final evaluation report is available here.

If you would like to learn more about the Cash Transfer Project, please see below for an informative blog and video:

Cash Transfer Program [English Version]

Helping a child grow: How cash transfers are helping children reach their full potential - Blog by Maria Svensson