Why And How Should We Measure Progress In Market Development?

In an earlier post in this blog series, Mayada El-Zoghbi  made a persuasive case for moving towards facilitating market development as compared to substantial direct investment in selected microfinance organizations. Alan Gibson then followed up with how donors can implement market development approaches towards increasing access to financial services.

My piece focuses on why should and how can we measure market development.

I come into this debate not as an evaluation expert but as someone who has observed and supported participation of the poor in different markets. The starting point obviously is why measure market development?

With increasing scrutiny of aid budgets in donor countries, the pressures to prove impact of development programs are higher than ever before. While many experts are focused on this, there is also growing acceptance that there is an equally important role of improving practice/ implementation, so that the projects/ programs can have faster feedback between implementation and lesson learning. In this way the impact of committed resources can be maximized.

However there is a third element which is highly important but in my view, much less understood. The process of prioritization, generation and dissemination of market data and insights is a critical part of market facilitation and development.

Finscope surveys in many countries illustrate how the participation of the diverse stakeholders in the process has triggered:

• Increased focus on levels of access and financial exclusion
• Informed debates amongst key stakeholders as to what can be done
• Convergence towards common vision/ road map
• Agreement on how to monitor progress and generate evidence for setting and tracking policy priorities
• Helping identify market opportunities for the private sector

The challenges of measuring impacts are even harder for market development programs for the following reasons:

• There is limited relevant market data in developing markets;
• A market-led approach to stimulating change in any development process does not result in any obvious ‘hard’ outputs that can be measured definitively and then compared with the costs of their production;
• A market-led approach provides a means to facilitate a process of change, but it cannot be expected to supply any detailed forward looking (ex-ante) statement of the precise activities and events that will eventually emerge from that facilitation process;
• An influencing agenda by definition involves other players, meaning attributing the credit for ‘success’ to any one or any group of players including donors would invariably be judgemental and may be difficult;
• The market development approach is based upon a theory of change that involves complex impact pathways operating both directly and indirectly, and at every level of the financial sector. The operations of these pathways are inherently complex and so difficult to evaluate.

It is relatively easy to assess impacts in a backward-looking fashion (ex-post) when there is already some record of progress and a strong narrative on which to build. A team at Oxford Policy Management (OPM)  undertook a study on FSD Kenya for the period 2005-10 to assess whether based on reasonable assumptions and attributing part of the outcomes to FSD Kenya, the value of benefits justified the investments made by donors. The analysis showed that even if only 2.83% of the quantified benefits were attributed to FSD Kenya, the benefits would still justify the donor contributions. Moreover, further research on non-quantified benefits is likely to significantly increase our understanding of the overall level of impact of the program.

The process of trying to quantify this value for money of FSD-Kenya generated many lessons that can be relevant for others wishing to explore how they can go about measuring the impact of market development programs. These include:

• In well-conceived projects, where some of the benefits are large enough on their own to justify the full costs of that project, we need not measure every output/ outcome;
• The results of market facilitation can be transformational (much greater than the sum of  its parts) but dis-aggregation is not easy e.g. TA to MFIs on its own would not have produced the results if the more enabling rules of the game were not facilitated;
• Outcomes are not always correlated with direct expenditure (many stakeholders noted that FSDK advice sometime was more valuable than the much larger investments made);
• The right advice on demand can deliver magnified results but assessing what would have happened without this advice is not easy;
• Success always has many parents (failure is often the orphan) and so there is a strong need to triangulate the evidence (to confirm attribution);
• There is a need to track short term trends and provide regular feedback to market actors but there must be a sufficient time horizon to assess the benefits arising from market facilitation;
• Measurement time and costs have to be proportionate to the program/ project costs (drive towards value for money of the measurement itself)
• There is a need to explicitly share data sources and assumptions for scrutiny by other stakeholders.

The measurement challenges are more daunting at the start of a market facilitation program, when impact assessment design needs to be forward-looking but when there is still great uncertainty as to where are the key blockages (rules of the game, support functions or core market)  and the likely success rates of different sub-projects. Building on the ex-post Value for Money analysis, the OPM team is now working with FSD Kenya to develop a monitoring system that takes into account this learning but allows for ongoing assessment of progress rather than waiting for completion of the programme to  undertake an ex-post evaluation.

-------The author is Principal Consultant, Financial and Private Sector Development at Oxford Policy Management, UK.