Formative Evaluation of the Out-of-School-Children Initiative (OOSCI)

Background:

The Out-of-School Children Initiative (OOSCI) was founded and launched in 2010. It aims to support governments to develop and apply innovative approaches to better estimate the number of children that are excluded from educational opportunities, identify who the children are, and to develop solutions to bring the children back to school.

OOSCI is a partnership between UNICEF, the UNESCO Institute for Statistics (UIS), and the Global Partnership for Education (GPE). GPE joined the initiative in 2013, and provided a grant to be used to create greater awareness around the issue of out-of-school children and to accelerate progress in achieving the outcomes of the initiative. The International Labour Organisation (ILO) and the World Bank supported the work of OOSCI through Understanding Children’s Work (UCW) . The work of the initiative is organised around three programme objectives, namely:

• To develop capacity and robust processes for deriving profiles of out-of-school children and to analyse barriers that have led to their exclusion;
• To identify and implement effective policies and strategies to reduce out-of-school children, and to integrate, the necessary changes within education sector plans [and thereby enhancing the likelihood for their sustainability]; and,
• To engender greater international attention and enhanced advocacy that will translate into commitments (national and international) to bring all children into school.

A formative evaluation of OOSCI was commissioned in 2017 to obtain an independent appraisal of the progress that governments have made to enact policies and implement solutions designed to reduce the number of out-of-school children. The evaluation aims to verify the contribution of UNICEF and partners in opening up learning opportunities - formal, non-formal or informal - for all children, and to enable the programme to meet its accountabilities to OOSCI donors. The evaluation covers the entire period of OOSCI implementation, from its inception in 2010 through the 2016 reporting period.

One of the activities that were undertaken by the UNICEF education team in preparation for the evaluation was to articulate a theory of change for OOSCI. The theory of change postulates that the provision of detailed data and evidence on why children are out of school, coupled with extensive advocacy efforts, will prompt governments to implement changes in their education systems that are necessary to bring children into school, and to achieve the stated goal of OOSCI - substantial and sustainable reduction in the number of children that are out of school.

Purpose, Objective, Scope and Use:

Purpose: The purpose of the evaluation was to test the validity of the programme theory of change and its assumptions, to provide a formative assessment of progress towards the achievement of the overall goal of achieving a substantial and sustainable reduction in the number of children that are out of school, and to strengthen the programme logic.

Objectives: Three objectives defined in the evaluation terms of reference were as follows:

• To examine the efficacy of strategies supported by UNICEF towards realising the goal of universal participation in basic education , and to determine whether pathways to reaching the intended goal are articulated clearly and are aligned with those of key-partners.
• Determine the extent to which OOSCI studies generated credible evidence on out-of-school children, influenced key policy changes, and facilitated the selection of effective strategies and interventions for various programming contexts, including countries undertaking humanitarian programming.
• To assess UNICEF’s contribution in building individual and institutional capacities to address barriers to entering and staying in school, assess their adequacy, and evaluate efforts at building capacities of key partners.

Organized around OOSCI programme outcomes, descriptive and normative evaluation questions are presented in the evaluation matrix in (see Appendix 3). The evaluation addresses the OECD/DAC evaluation criteria of relevance, effectiveness, efficiency and sustainability as promulgated by the Organisation for Economic Co-operation and Development's Development Assistance Committee (OECD/DAC). It also addresses two additional criteria; coherence, to enable assessment of the formative aspects of OOSCI and the evolving nature some concepts and tools, and, utility.

Scope: The evaluation covered all OOSCI partner countries that had completed their studies by the end of 2016, estimated at 40 of 87 countries. Countries were spread through all UNICEF regions. Partner countries are at different stages of OOSCI implementation (conducting their studies, policy level work, etc). The evaluation also covered inputs and activities of OOSCI core partners, namely UNICEF, UNESCO Institute for Statistics, the GPE, and respective governments.

Evaluation use: The evaluation was meant to facilitate reflection and learning among education managers responsible for programming on out-of-school children issues in all participating agencies. A possible revision of implementation strategies aimed to improve programme coherence is anticipated. Policy-makers and government counterparts are expected to use evidence from the evaluation to deepen their understanding of the issues facing out-of-school children at all levels of the education system, as well as to mobilize stakeholders in key sectors, such as the social services sector.

Methodology and Approach:

A theory-based design was employed for the evaluation, with the OOSCI theory of change being articulated retroactively by OOSCI managers during the scoping phase of the evaluation.

Evidence of the contribution of UNICEF and that of partners was derived through a qualitative design. Sources included (i) a desk-based review of secondary data analysis; (ii) an online survey, administered to education programme officers in all UNICEF country offices implementing OOSCI; (iii) interviews and focus group discussions with a sample of respondents in UNICEF New York, Regional Education Advisors and/or OOSCI Focal Points in all seven UNICEF regional offices; and (iv) interviews and focus group discussions held with stakeholders at country level during the course of the field visits.

All primary and secondary data were subjected to a qualitative content analysis and/or a qualitative comparative analysis (QCA).  QCA was used to answer evaluation questions relating to different contextual conditions under which OOSCI was implemented, and combinations of factors that would make the reduction of the number of out-of-school children more likely in one context, and less likely in another. Finally, survey data was subjected to descriptive analyses (e.g. mean values, standard deviations). These analyses were mainly used for triangulation with an additional data source, and to substantiate the qualitative findings with a larger empirical base.

Findings and Conclusions:

Selected findings Progress towards universal basic education

• Declarations of universal basic education, expressed or implied, were found in government documents for the majority of OOSCI partner countries (80 percent), signalling a strong intent to eliminate the problem of children being out of school.
• OOSCI is credited with having brought a positive attitudinal change to government partners on the subject of out-of-school children, and with bringing new energy and a new push for prioritizing programmes on issues facing out-of-school children in UNICEF country offices.
• With nearly 70 percent of the countries having executed or completed an OOSCI study, the Middle East and North Africa region (MENA) maintained an intense focus on OOSCI and issues relating to out-of-school children. These efforts were beginning to show dividends, both in terms of targeting approaches, and the variety of solutions for out-of-school children.

However,

• Many OOSCI countries often conflated “inclusive education” with special education programmes, and because of this lack of conceptual clarity, interventions failed to address exclusion of specific groups of out-of-school children.
• The link between stated goals for universal basic education, its objectives, and proposed and/or implemented strategies was often inconsistent, and sometimes contradictory.
• Also, sub-national authorities often lacked the data and/or evidence required to make a strong push for investing in education, and/or to devote the necessary time and resources to dedicate to inclusion strategies.

Evidence generation and utility of OOSCI studies

• OOSCI studies executed in partner countries were found to be effective in generating profiles of children that are out of school, and in identifying barriers that prevent children from enrolling in school, cause them to drop out, and/or prevent them from re-entering school appropriately.
• OOSCI has contributed positively to increasing the visibility of the subject of out-of-school children, and the plight of the children that are excluded from participating in school in the education development discourse, policy dialogue, and in priority setting agenda.
• Where countries were not successful in generating complete profiles of out-of-school children or in identifying and addressing barriers that keep children from school, a reduction in the number of out-of-school children were still realized, provided that the countries were relatively prosperous (using a high human development index as proxy for prosperity) and were judged as stable (i.e., Fragile State Index of 60 or less).
• Almost all OOSCI studies were successful in coming up with robust data, and where possible, estimates of the number of children that are out of school; however, these figures were often contested, resulting in some countries being reluctant to release their studies for public consumption.

However,

• The “five dimensions of exclusion” as articulated by OOSCI were not adequate to describe all profiles of out-of-school children. The evaluation also found that the upper-secondary school population should be included in the OOSCI methodological framework in order to make it more responsive to the different country contexts.
• Solutions for eliminating the barriers that keep children away from school were not a key component of the priority setting agenda in most OOSCI partner countries.
• While OOSCI studies were successful in generating recommendations to address key issues affecting out-of-school children, the recommended actions were sometimes tenuous in terms of addressing the most prevalent barriers and bottlenecks, and at times not feasible and/or actionable.

Partnerships to advance the work of out-of-school children

• OOSCI partnership arrangements and the division of tasks between the core partners were considered to be cohesive, productive and to have increased efficiency for the majority of implementers, while the contribution of all OOSCI partners was credited for having expanded geographical coverage of activities and interventions for out-of-school children. This outcome was highly valued by participating governments.
• OOSCI is credited with a creating a higher demand for technical and policy advice around issues affecting out-of-school children, and to have increased opportunities for face-to-face interaction with decision-makers.

On the other hand, the evaluation also found that:

• OOSCI non-government partners were not diverse enough. National civil society organizations were underrepresented in the work of out-of-school children in comparison with international non-governmental organizations (INGOs). Also, the roles and results expected from this category of partners were not clearly defined.
• OOSCI was highly valued by smaller NGO partners whose views are rarely represented in policy debates, and who regarded the opportunity to work alongside OOSCI as reclaiming their “voice”.

Strengthening education systems and capacities Derived from Chapter 6, the findings on strengthening of education systems and capacities signal that the overall sustainability of OOSCI objectives and government efforts are shaky, at best. For instance:

• Half of the countries sampled for the document review demonstrated only modest success in improvement of data systems and processes, while commendable success was registered in only in a small number of countries.

On the other hand, the evaluation also found that:

• Availability of robust and reliable data was highly inconsistent, due mostly to limitations in financial and human resources capacities for data collection, data analysis, data interpretation, and related processes; and,
• A robust contribution analysis to explain the factors that account for reductions in the number of out-of-school children is both methodologically possible, and necessary to sustain the evaluability of OOSCI.

Conclusions Conclusion 1: Universal basic education is still a unifying goal and message for what the education sector is required to achieve in terms of maintaining high enrolment, retention, and completion rates. Beyond these measures of participation and efficiency, UBE is increasingly being reconceptualized to include equity and inclusiveness, which also means that education resources should be allocated to achieve progressive universalism. Adopting a formal definition of UBE to reflect this thinking would strengthen the linkages between the objectives of OOSCI, UNICEF’s advocacy and resource mobilization efforts, and other work around out-of-school children, as well as the overarching goal of improving education outcomes for all children.

Conclusion 2: OOSCI studies have laid an important foundation in developing comprehensive profiles of out-of-school children in each country, and in identifying barriers. The analysis of barriers needs to be contextualized and updated periodically in order to remain to responsive to the needs of different groups of out-of-school children.

Conclusion 3: Evidence and policy guidance from OOSCI studies have become a useful resource for planning processes in education departments and for education sector partners. To the extent that the initiative has gained acceptance in the partner countries, OOSCI is well positioned to push important messages (such as the value of stability in terms of a lack of conflict, and a productive economic environment), and to provide support to turn those messages into action that is tailored for different programming contexts.

Conclusion 4: While the contribution of UNICEF and OOSCI partners has led to discernible progress and changes in policies and planning, a gap between policy and planning on one hand, and implementation on the other remains, due mainly to inadequate prioritization of issues facing out-of-school children. The evaluation concluded that a new advocacy effort for the out-of-school children agenda is required. So is the prioritization of solutions and/or interventions for the most disadvantaged sub-groups of children that are out of school, as well as a resourcing model for issues facing all children that are out of school.

Conclusion 5: In an operating environment subject to frequent changes in government staffing decisions, shifting donor resources, and continuous movement of people, UNICEF was regarded by all actors as a constant factor, and a reliable “anchor partner”; its convening power helped to move the partnership objectives forward.

Conclusion 6: Technical capacities to identify and serve all children, including all profiles of children that are excluded from school, were strengthened. However, improvements were confined to individual capacities, and did not permeate the system. As such, the gains from OOSCI will not be sustainable in the long run, unless the next generation of OOSCI studies concentrate greater effort on supporting governments to achieve systemic changes.

Evaluative assessment Relevance: OOSCI was found to be relevant to national and international debates on equity in development. By highlighting the plight of out-of-school children, even as countries celebrate gains in enrolment rates and progress towards the MDGs/SDGs, it has raised issues of equity and fairness as well as the rights of children, in the quest to make full use of the human resource potential of countries. In addition, through policy dialogue and strategic support, OOSCI has enhanced its relevance in helping to shape national priorities and to formulate robust sector plans that embrace education as a right for all children.

Effectiveness: In most countries, OOSCI was effective in cultivating a critical mass of national stakeholders who are ready to support the shift from targeted community interventions to an effective systemic approach, with regard to out-of-school children. Consequently, an effective and inclusive process of policy making and priority setting around out-of-school issues was triggered at the macro level. However, OOSCI was less effective in supporting countries to translate recommended policies and strategies into concrete practice.

Efficiency: By being embedded in priority setting processes, developing sector plans, and mobilising resources; OOSCI partner agencies have contributed efficiently to measures that address key challenges posed by the problem of out-of-school children in target countries. Efficiency could be improved by keeping all OOSCI partners engaged by assigning more roles and tasks, and through deeper collaboration between OOSCI partners. This would enable partners to better “deliver as one” in providing their support for measures that help to translate policies, plans, and priorities into concrete achievements on issues pertaining to out-of-school children. In this regard, marshalling research capacities of a partner such as UCW and allocating specific tasks to the group should increase efficiencies of OOSCI.

Utility: The utility of OOSCI is closely tied to its effectiveness, Useful outputs have been put in the hands of governments. As an initiative designed to support addressing key challenges and reducing the number of out-of-school children in the population, the utility of OOSCI is also linked to availability of resources on a sustainable basis. Without this, the problem of out-of-school children will persist or worsen, no matter how many studies and strategic plans the initiative generates for any given country.

Coherence: Barriers to universal education are complicated and intertwined. Hence, the ability to deliver a comprehensive national, regional, and global response depends on sound interrogation of concepts and claims about what OOSCI can deliver. In that regard OOSCI was internally coherent enough to be functional it its formative phase. As end users begin to expect more of OOSCI, additional work will be required to make it conceptually sound, and coordinate effectively across sectors and among stakeholders. OOSCI’s external coherence was also low, due to weak cross-sectoral coordination and failure to attract the necessary non-traditional partners. There is still a need to improve coordination and strengthen leadership on programming, to seek out and engage with less prominent by significant partners.

Sustainability: Sustainability depends not only on resources (or a lack thereof) at country level, but also on the political will and commitment of governments and partners to a rights-based model of education. Resources facilitate the implementation of feasible solutions, and commitment drives efforts towards progressive realisation of the goal of basic education for all. Both resources and commitments are not yet at levels that would make for sustainability in addressing the challenges posed by out-of-school children. This is particularly the case for domestic resources, implying a need for long term external support.

Recommendations and Management Response:

Recommendation 1: The theory of change for OOSCI should be revised to reflect the key elements of inclusion to ensure that the needs of all out-of-school children are met at all levels of the basic education cycle, while still ensuring that the initiative focuses on identifying strategies and policies that empower governments to eliminate the lack of participation at the pre-primary level, such as sustainable, pro-poor financing for the sub-sector. Management response: UNICEF management agrees to revise the theory of change for OOSCI to incorporate all levels of education from pre-primary to upper secondary while still maintaining a special emphasis on the crucial early years of education. An expert group consisting of staff from UNICEF HQ, regional offices and country offices, as well as partners such as GPE, UIS and ILO will be established to revise the theory of change, new tools and methodologies for developing profiles of out-of-school children.

Recommendation 2: OOSCI should expand its focus to harness the expertise and capabilities of OOSCI technical partners to seek effective and efficient strategies and solutions that support the implementation and comprehensive monitoring of policies in key contexts where programming for different profiles of out-of-school children occurs, and to attract resources to ascertain sustainability of implementation. Management response: UNICEF management agrees that OOSCI should focus on strategies to support policy implementation and monitoring, and engage technical partners in this work. Once the theory of change has been revised, the expert group will develop and circulate guidance for conducting OOSCI country studies including on strengthening links with existing processes such as the development of Education Sector Plans meet the needs of out-of-school children.

Recommendation 3: OOSCI should re-orient its methodological framework towards the entire basic education cycle (i.e., pre-primary to upper secondary), and target key vulnerable groups that cut across all profiles of out-of-school children, and generate explicit strategies that address the learning needs of these groups, including but not limited to embracing appropriate forms of learning for them, and responsive modalities for delivering those learning opportunities. Management response: UNICEF management agrees that the OOSCI methodological framework should be expanded to include adolescents of upper secondary school age and other forms of education outside formal schooling such as alternative and flexible education programmes. The expert group will develop the tools to create statistical profiles of adolescents who are Not in Education, Employment or Training (NEET), and guidance on incorporating non-formal education in the OOSCI analysis.

Recommendation 4: While maintaining the usual focus on supporting governments to discharge their mandate to extend learning opportunities to all children, OOSCI should facilitate processes for assembling the right type of partners, including but not limited to government officials, that have a clear potential to bring new ideas, and/or offer new entry points for programming for out-of-school children. Management response: UNICEF management agrees that it is essential to continue to build coalitions and partnerships to respond to emerging issues and changing situations, including greater engagement by civil society, multi-lateral and bilateral agencies, religious organizations, the private sector and non-traditional donors. UNICEF (Education Section) will commission research and issue briefs on engaging with other service providers including low-cost private schools and Quranic schools, and encourage Country Offices to actively promote the inclusion of civil society and other stakeholders in Local Education Groups that advise partner governments and advocate for specific groups of marginalized children.

Recommendation 5: OOSCI should strengthen all its programmatic elements to set the initiative up to yield evaluable information on the stated goal of achieving a substantial and sustainable reduction in the number of out-of-school children. This includes ascertaining the internal and external coherence of the initiative, the feasibility of achieving intended results, and ensuring that adequate M&E inputs and systems are put in place to enable systematic assessments of OOSCI’s contribution. Management response: While the formative evaluation has demonstrated that OOSCI has made a valuable contribution to efforts to reduce the number of children out of school, UNICEF management agrees that the expert group also establish indicators and monitoring arrangements to collect both quantitative and qualitative data that will enable the progress of the initiative to be systematically assessed in the future, both in development and humanitarian contexts.

You can find here the OOSCI infogram.