Where The Wild Things Are: Measuring Market Development Progress

He sailed off through night and day
And in and out of weeks
And almost over a year
To where the wild things are.
--- Maurice Sendak

Immediately prior to this year’s European Microfinance Platform held in Luxembourg, CGAP convened a group representing donors, development finance institutions (DFIs) and experts in finance and market development to discuss the evolving role of donors and DFIs in advancing financial inclusion.

The workshop opened with a discussion of the market development approach (also referred to as “market systems development” and “M4P” – short for “making markets work for the poor”) and its application to financial markets. (See a brief summary of this topic in a recent post by Alan Gibson of the Springfield Centre here.) Some of the workshop participants are already incorporating a market development approach into some aspects of their private sector work although only one – the U.K. Department for International Development – consistently applies the M4P approach to its financial inclusion work.

One of the sessions of the workshop addressed the question that all donors and DFIs ask when considering taking a market development approach: how does one measure progress? (The end goal of a market development project is to create a market that serves the poor and that will, after a certain point, no longer require donor funding or subsidy.)

Being able to measure and demonstrate progress are critical issues as taxpayers and other funders of donors and DFIs require explanations and justifications of the use of their limited resources. A handful of donors are starting to experiment with different tools for measuring progress in agricultural and other non-financial markets. It is difficult work, especially when attempting to measure attributable and sustainable changes to a market system.

Sukhwinder Arora, one of the speakers on this session, summarized OPM’s work last week in his blog on measuring the impact of FSD-Kenya.

Jim Tanburn for the Donor Committee for Enterprise Development (DCED) was also on this session and began his talk with the main challenges to measuring system change. He underscored that there is a world of difference between a donor’s efforts to change a system (where the donor’s role may change over time) and an isolated project, such as introduction of a new financial product or service. And likewise, there is a world of difference between measuring system change results (especially if the donor’s role changes over time) and measuring the results of a new product.

As Jim noted, although markets de facto change over time in response to various internal and external dynamics, a donor needs to measure progress of a specific intervention (“how are we doing? Are we having the results we expected?”) in real time. This requires good monitoring based on a well-developed theory of change, which requires strategic clarity.

Monitoring must be designed to measure the right data and must be adjusted in response to changes in the project design, some of which may be implemented in response to the data gathered through monitoring. (Jim commented that while the OECD/DAC-supported Working Party on Aid Effectiveness states that good monitoring involves the “systematic collection of data on specific indicators,” many implementers are not yet doing this.) A theory of change requires that a donor articulate the expected outcomes of an intervention before implementation. This is difficult to do, given that a system change affects the system as a whole, making it difficult to measure and isolate what would have happened without the intervention.

A usable theory of change must necessarily simplify a complex system into a generally linear representation. Jim noted that when implementers are pushed to articulate and validate the assumptions behind their theories of change, they often uncover gaps in their logic. To assist donors in their efforts to develop and use theories of change, DCED has developed a framework (the DCED Standard).

While there are significant challenges to measuring market development, monitoring and measurement are – as Sukhwinder wrote in his blog – part of the process of making markets work for the poor and not just a tool to see the progress of a specific project. As this is the objective of those involved in financial inclusion, I would suggest (quoting Maurice Sendak again) that we “let the wild rumpus start!”

---- The author is a lawyer and a policy advisory consultant to CGAP.