USAID’s Approach To Developing Market Systems
USAID, I imagine like other donors, is a diverse and complex Agency. Within this complexity, there are nodes of people who are thinking in more systemic terms. Within its Economic Growth technical area, which is where financial systems are located, there are some people working on advancing an agenda focused on market systems. USAID’s learning bureau is also engaging in discussions and research on systems. They too are interested in how to design and evaluate all kinds of systems.
Developing an effective approach to evaluation is key to furthering a systems approach. USAID and others are approaching this topic of evaluating systemic impacts. They have delved into it to some extent, but no one – that I am aware of – has a very developed or definitive answer for how to measure systemic impacts.
Why is this? Most of us agree that traditional experimental approaches to evaluation taken alone are ineffective for evaluating market – including financial - systems. There are at least three reasons for why market development calls for a different approach to evaluation: the market system perspective, the facilitation approach to implementation, and project evolution.
Market systems are dynamic and changes stimulated by projects are rarely linear, making it difficult to match each activity with a single output and outcome. Rather, activities can generate multiple outputs, and outcomes are interconnected and can occur at different points in the market system. Moreover, traditional M&E approaches tend to emphasize baseline and end line indicators such as volumes and values of production, productivity, income. However, we found that changes in relationships, interactions and behaviors of market actors were much better indicators of sustainable change. Yet these indicators are more challenging to capture.
To further complicate matters, market development projects use a facilitation approach to implementation. In seeking to catalyze improvements, facilitation ensures that the actors themselves drive the change process. Facilitators work indirectly with market actors. As a result, a project does not control who participates and who does not. This creates difficulties for impact evaluations that seek to randomize the treatment group that receives project services and to create a control group.
Lastly, market development projects are dynamic. Because they intervene in dynamic and often unpredictable market systems, interventions often shift over time to reflect the shifting needs and priorities. Good market development projects evolve over their lifetime as they seek to adapt to shifting environments and seize new opportunities. Project evolution can make parts (or even all) of a baseline survey irrelevant. This is challenging for a funding agency that expects results to be achieved following the original project plan.
USAID has been exploring ways of addressing these challenges to measuring systemic impacts. We have written a paper on an approach we have called Degrees of Evidence. It is not a methodology rather an outline of a comprehensive approach that calls for mixed methods – experimental, non-experimental, quantitative and qualitative – with the intent of developing a preponderance of evidence regarding impact and attribution. It underscores that there is not perfect evaluation methodology; even RCTs which are considered the gold standard have weaknesses. The degrees of evidence approach aims to balance the strengths and weaknesses of a variety of methodologies and to triangulate findings to ascertain a preponderance of evidence regarding impacts and attribution.
But as an Agency and as a community, we are still at a nascent stage and are planning to further develop this thinking through collaboration with the nodes of innovation within the Agency. A lesson in developing this new and exciting area – systems development and evaluation of systems – is the importance of investing in a focused program or programs for knowledge generation, while engaging collaboratively with others in the Agency that are also innovating and open to learning.
Adopting a systems approach within a large bureaucracy requires a cultural shift. Such a shift only comes with persistence, high-quality knowledge and collaboration that expands over time.
Also in this Series
New Directions for Funders: Insights and Perspectives
How can funders balance the need for flexibility and accountability requirements with partners while ensuring effective outcomes? The answer lies in defining a monitoring and impact evaluation system that includes a "theory of change," quick feedback loops and reasonable assumptions.
Market Building Through Financial Sector Deepening In Africa
How do we support a market building approach in practice? The current advice is to either become a market facilitator or fund one. But what is a market facilitator; what do they do?
Comments
Dear Jeanne. Thank you so
Dear Jeanne. Thank you so much for this post. It is a great pleasure to read about the things that are going on within USAID. The thinking and insights that you describe are very much in line with my experience working on very similar issues but from a bottom-up perspective. I had the pleasure to work in the SEEP Network's recent Initiative called 'Systemic M&E' that brought together practitioners in market and financial market development with experts from the fields of complexity sciences and systems thinking to discuss ways to improve the current M&E paradigm. Some ideas of these discussions also went into my recent blog-post: http://marcusjenal.wordpress.com/2013/01/24/bottom-up-perspective-on-re…
I am looking forward to more healthy discussions around this topic and hopefully soon also concrete changes in the field.
Thanks Jeanne! It's great to
Thanks Jeanne! It's great to see USAID being increasingly involved in leading dialogues around new paradigms in evaluating market systems development.
Two of the best interviews / presentations that I've heard on this were by Dave Snowden and Shamin Bodhanya who talk from a complexity sciences perspective and how this links to our market development field of practice. The links to these are below.
http://www.seepnetwork.org/systemic-m-e-discussion-series--an-interview…
http://www.seepnetwork.org/systemic-m-e-discussion-series--an-interview…
Jeanne, thanks for this great
Jeanne, thanks for this great blog. It's so encouraging to see USAID investing in this important evolution of development practice. I have been in Nairobi this week and the term 'systems approaches' is being used by many which is great, but I also understand that there is little evidence of implementation. However we are in the early stages of this shift. I am sure that your paper(which I must read!) and other work like the SEEP/MAFI practitioner initiative will create a tipping point in development thinking and practice.
All the best
Alison
This is a refershing
This is a refershing perspective of M&E that starts to incorporate mutual and multiple causality, and the adaptive nature of complex systems such as markets Thank you.
Dear Jeanne,
Dear Jeanne,
So happy to come across this … Am having a hard time explaining the outcomes of our recently concluded value chain dev project and that it cannot just be viewed like an enterprise dev project under a direct intervention approach.. The article describes more or less the issues we are facing… and, as usual, we made modifications on the interventions based on assessment of the situation.
Will search for the paper “Degrees of Evidence”.
Dear Jeanne,
Dear Jeanne,
Thanks for your (and USAID's) insightful views on the subject of evaluating systemic impacts of market development initiatives.
1. Has there been an examination by USAID of the pros and cons of the usefulness of the DCED Standard for measuring results in PSD and, if so, what are the results?
2. I agree with you that a cultural shift is probably the most important issue to tackle. Such cultural shift does not only need to take place in large bureaucracies, but also with the general public, where scepticism about development impact is rising. At the end of the day there may be a bigger challenge in making the general public and politicians aware of another way of looking at systemic change in market systems. Presently there appears to be much more focus on easy answers in terms of quantitative and tangible results, than on furthering innovative systems approaches. Without getting the general public and politicians on board development practitioners may risk falling victim to the law of the handicap of a head start.
Your post is a reflection of
Your post is a reflection of the leadership you have provided at the USAID Global Bureau’s Office of microenterprise development. You have been at the helm with USAID from BDS and targeting individual businesses, to bringing in the value chain approach so that there would be more impact. And yes, our field may have started as a linear approach with systematized steps, but with your leadership, resources were dedicated through AMAP to do some research on such a myriad of issues like market development and conflict, HIV-AIDS, putting funds towards the results measurement and getting some data on real projects.
Heck it has been a ride over the last 10-15 years. Your post indicates that USAID is responding to the on the ground learning—and that systems thinking requires new ways of responding to M&E—and this is not just for donors, it is for us to be better facilitators and managers and to build human capital with our field staff so that they also understand that markets are dynamic and changing-- Orthodox economics are still holding on to static equations to understand how the economy grows and shrinks, and we as a field working with the majority of people in the world are on the cutting edge of really getting at being flexible and responsive to complex realities. It is exciting times— kudos to you and thanks for your thoughtful leadership!
You go girl!
Cheers
mary
Dear Jeanne, thank you for
Dear Jeanne, thank you for the concise update of the thinking in USAID. It is important that the softer evidence, like changes in relations, behavioral changes and even attitudes be measured or at least be recognized as signal of change.
Best wishes,
Shawn Cunningham
Thank you very much Jeanne,
Thank you very much Jeanne, for describing this refreshing search for a new approach to impact assessment for market development interventions.
For too long and in the face of growing evidence of inadequacy, many people have advocated an overly mechanistic and numerical approach to impact evaluation. Typically relying on elaborate spreadsheets full of assumptions (that are then frequently multiplied by other assumptions), very precise, but ultimately unverifiable, claims of attributable results have been made.
This over-simplification (sadly reflected in the thinking behind the DCED Standard) has two major failings:
1. The numbers generated mean nothing in the real world so do not communicate a credible story to the people we need to convince (something that I think Leon is alluding to in his comment); and
2. The numbers miss so much of the wealth of impact that a successful market development intervention can generate, so ironically under-report development achievements.
I sincerely hope that the USAID team working on this issue manage to reconcile corporate public accountability requirements with the complexity of the interventions and impacts that they seek to measure. This is probably no easy task, but the enthusiastic response to this post shows quite how much their efforts are appreciated by many others in the development world - particularly those who are implementing market development projects and see the impressive results that such interventions can achieve.
Jeanne, Congratulations on
Jeanne, Congratulations on taking this important step to develop USAID's understanding of evaluation in complex systems. I look forward to opportunities to apply this work to USAID's youth workforce development programs! Rachel
Dear Jeanne, it is refreshing
Dear Jeanne, it is refreshing to see that USAID is struggling with many of the same evaluation issues that the CGIAR is facing regarding value chains. In addition to suggesting that a connection between your work and the Policies, Institutions and Markets (PIM) program of the CGIAR, you might also find the recently published work on assessing the relationships between chain actors from IIED et al to be of use. You can access their method on measuring fairness among supply chain participants at http://pubs.iied.org/16042IIED.html? and the survey instrument itself at http://pubs.iied.org/G03429.html? We are planning to test this approach in Colombia this year and would be happy to share some results towards the end of 2013.
Jeanne, thanks for your
Jeanne, thanks for your update on the current thinking within USAID. Systemic change is really where the big impact of PSD can lie, so exciting to see more resources put into the considerable challenge of assessing it.
I would certainly agree with James below on the risk of producing meaningless impact figures - I'm sure we've all seen this happen in practice. However, the DCED Standard does make it clear that any quantification of impact needs to be supported by reasonable assumptions and, a broader understanding of what they mean. Numbers certainly aren’t everything.
The (recently redrafted) DCED guidelines contain several sections on systemic change, qualitative indicators, and flexibility that aim to help move the debate past a simple dichotomy of ‘over-simple and quantitative’ vs ‘complex and unquantifiable’. See http://www.enterprise-development.org/page/measuring-and-reporting-resu….
Jeanne - thanks for sharing!
Jeanne - thanks for sharing! The Market Systems approach to financial sector development through facilitation offers a more effective use of funds and potentially higher sustainable return on investment. But, as you rightly state, it becomes more complicated to evaluate, measure, and attribute impact for those donors funding them. I appreciate USAID’s leadership in supporting the emergence of an approach to addressing this challenge. The Financial Sector Deepening Trust in Kenya (FSD Kenya) offers an interesting case study on how to measure a dynamic, multi-year market systems program. They recently commissioned a study to conduct a range of reviews and evaluations of FSD’s activities, covering the period since its establishment in 2005. While the study was able to provide a positive evaluation of FSD’s achievements against its objectives, it also provided some thoughtful perspectives on the difficulties of assessing systemic initiatives using traditional methods of project evaluation. Perhaps you have already seen this study, but I figured I would share to you and the wider readership.
http://www.fsdkenya.org/pdf_documents/OPM_Working_Paper_Assessing_Value…
Thanks for this post. I
Thanks for this post. I wanted to ask if anyone has any recommended readings on market systems within the context of value chains and value chain promotion.
Add new comment