How to Measure Results of Financial Sector Development Programs

For more than a decade, some funders of financial inclusion have been investing in market development approach programs as a part of a strategic shift. For financial inclusion, the most known of these organizations are the Financial Sector Deepening programs (FSDs) – multi-donor entities that strategically invest through grants, technical assistance and research. Yet funders are still largely working on a belief that market facilitation, rather than specific support to financial institutions or regulators, does more to expand sustainable access to financial services for the poor.

With limited attempts at measuring the impact of investments focused on market development,  it is difficult to conceptualize how financial markets in places such as Kenya, Tanzania, and Nigeria would have developed without support from FSD programs, and whether the impact of market development approaches is sufficiently different from traditional funding approaches. This knowledge deficit in part reflects the relative newness of this approach but also reflects the measurement challenge of assessing changes in complex market systems.

Oxford Policy Management (OPM) and CGAP have been exploring how to address these challenges, based on work with FSD Africa (FSDA) and the eight established FSDs in Africa.

No single methodology is likely to respond to all the measurement challenges this presents. However, a combination of the following methods and mindsets may collectively help provide a credible narrative for how a financial market changed and why.

Moving beyond measurement for accountability only: Measurement frameworks can and should be used for more than just reporting to funders. Market development programs should use the results to adapt to changing markets, scale up what works, and de-prioritize what does not work. They can also use them as a powerful mechanism for influencing market actors, regulators and policy makers, who are interested in tracking financial inclusion objectives and trends as well as policy implications. Even financial institutions have an interest in the measurement results to identify business opportunities with underserved customers.

Measuring number of people reached and system change: Funders often want to see impact in ‘numbers’ – for example the number of accounts opened as a result of their interventions. FSD programs need to better highlight how their influence on the underlying system can lead to improved access. For example, when working with regulators, programs can track how and why financial institutions have responded to a change in the regulatory space, and how this has led to more or less financial access. But programs also need to place increased financial inclusion numbers within the context of broader market changes. For example, rather than only reporting that mobile money accounts have increased, FSDs should measure if system dynamics indicate whether these are likely to be sustainable and resilient: Have regulators’ attitudes, behaviours, and practices to mobile money changed? Are new forms of information sharing and partnerships amongst mobile money organisations and banks occurring?

Embedding learning and adaptation within theories of change: FSD staff should regularly revisit their theories of change (ToCs), and these reviews should draw on the skills of technical staff as well as M&E specialists. This helps to ensure that the focus is not solely on compliance to the ToC, but also on learning and tweaking as the program evolves. Resources to establish baselines should be used flexibly, especially as heavy investment in baselines before proof of concept may be of limited use for changing markets and programs. Finally, a learning culture should be incentivized where failure and adaptation are encouraged by program management, as well as by funders –this last point is often agreed but rarely practiced.

Capturing unintended impacts and the complexity of market change: Markets are complex and unpredictable, but rather than ignoring this reality and limiting measurement to simple input- output-outcome linear chains, FSDs need to embrace such complexity, being humble about how much they can understand, predict, influence, and claim. This means FSDs need both program and non-program monitoring and evaluation (M&E) systems. A non-program perspective seeks to understand how the broader system is changing regardless of the program’s interventions e.g. the expansion of financial services by non-supported financial institutions, or if there are new market players disrupting the system.

Financial system change is complex, but the ongoing work recognizes that this should not divert focus to what is measurable but instead persist with measuring what is important. This work recognizes measurement as an essential tool for program implementation to learn from and adapt programs. The techniques noted above also provide FSDs with a more nuanced and realistic evidence base, providing independent external evaluators to confirm accountability to funders, but also helping funders and market actors prioritize investments in the future. 

Sukhwinder Arora and Rich Williams both work for Oxford Policy Management UK. They are currently working with eight FSDs in Africa, FSD Africa, CGAP and other OPM colleagues to facilitate agreement on possible approaches for measuring results for the financial sector development programmes.