USAID’s Approach To Developing Market Systems

USAID, I imagine like other donors, is a diverse and complex Agency.  Within this complexity, there are nodes of people who are thinking in more systemic terms.  Within its Economic Growth technical area, which is where financial systems are located, there are some people working on advancing an agenda focused on market systems. USAID’s learning bureau is also engaging in discussions and research on systems. They too are interested in how to design and evaluate all kinds of systems. 

Developing an effective approach to evaluation is key to furthering a systems approach.  USAID and others are approaching this topic of evaluating systemic impacts.  They have delved into it to some extent, but no one – that I am aware of – has a very developed or definitive answer for how to measure systemic impacts. 

Why is this?  Most of us agree that traditional experimental approaches to evaluation taken alone are ineffective for evaluating market – including financial - systems. There are at least three reasons for why market development calls for a different approach to evaluation:  the market system perspective, the facilitation approach to implementation, and project evolution.

Market systems are dynamic and changes stimulated by projects are rarely linear, making it difficult to match each activity with a single output and outcome. Rather, activities can generate multiple outputs, and outcomes are interconnected and can occur at different points in the market system. Moreover, traditional M&E approaches tend to emphasize baseline and end line indicators such as volumes and values of production, productivity, income.  However, we found that changes in relationships, interactions and behaviors of market actors were much better indicators of sustainable change. Yet these indicators are more challenging to capture.

To further complicate matters, market development projects use a facilitation approach to implementation.  In seeking to catalyze improvements, facilitation ensures that the actors themselves drive the change process. Facilitators work indirectly with market actors. As a result, a project does not control who participates and who does not.  This creates difficulties for impact evaluations that seek to randomize the treatment group that receives project services and to create a control group. 

Lastly, market development projects are dynamic.  Because they intervene in dynamic and often unpredictable market systems, interventions often shift over time  to reflect the shifting needs and priorities. Good market development projects evolve over their lifetime as they seek to adapt to shifting environments and seize new opportunities. Project evolution can make parts (or even all) of a baseline survey irrelevant. This is challenging for a funding agency that expects results to be achieved following the original project plan.

USAID has been exploring ways of addressing these challenges to measuring systemic impacts.  We have written a paper on an approach we have called Degrees of Evidence.  It is not a methodology rather an outline of a comprehensive approach that calls for mixed methods – experimental, non-experimental, quantitative and qualitative – with the intent of developing a preponderance of evidence regarding impact and attribution.  It underscores that there is not perfect evaluation methodology; even RCTs which are considered the gold standard have weaknesses. The degrees of evidence approach aims to balance the strengths and weaknesses of a variety of methodologies and to triangulate findings to ascertain a preponderance of evidence regarding impacts and attribution.

But as an Agency and as a community, we are still at a nascent stage and are planning to further develop this thinking through collaboration with the nodes of innovation within the Agency.  A lesson in developing this new and exciting area – systems development and evaluation of systems – is the importance of investing in a focused program or programs for knowledge generation, while engaging collaboratively with others in the Agency that are also innovating and open to learning.

Adopting a systems approach within a large bureaucracy requires a cultural shift.  Such a shift only comes with persistence, high-quality knowledge and collaboration that expands over time.