Evaluating Financial Literacy for Branchless Banking Clients

IFMR-Research recently conducted an impact evaluation of the financial literacy program delivered to EKO clients in the northern Indian state of Bihar. EKO provides mobile banking services such as savings and withdrawal services, to clients via Customer Service Providers (CSPs) operating in urban and rural centers in India. The evaluation funded by the World Bank, assessed the impact of exposure to an intensive one hour training on clients’ savings behavior; on topics such as the importance of savings with formal institutions, developing savings goals based on their long-term and short-term life needs, understanding the range of savings products available for disparate savings needs and general expense and income management.

Besides evaluating the impact of financial literacy, a series of process evaluations were performed to determine the effectiveness, relevance, and feasibility of these financial literacy sessions.

All critical Financial Literacy training processes and sub-processes were identified, as indicated in the following Process Map:

1. Trainers Training Process - IFMR provided trainers a 10-day training (called TOT – Training of the Trainer). Trainer interviews were conducted via phone to evaluate the TOT process. Questions about trainer experience during IFMR training sessions, relevance of the training materials, effectiveness of training materials used by CMF, feasibility of training locations and timings were asked.
2. Client Training Process – Trainers conducted a one hour interactive financial literacy session for clients grouped in clusters of 10 clients on average. This process was evaluated by conducting interviews with 10% of clients randomly selected from each cluster. The clients were asked about their experience during the training, feasibility of training locations and timings, relevance of training sessions in their day-to-day lives, and suggestions for improvement of the overall training delivery mechanism.
3. Client Knowledge Acquisition Process – Clients were provided comprehensive financial literacy during the sessions. Clients’ knowledge about various financial topics was evaluated by conducting interviews with 10% of clients randomly selected from each cluster. To determine if trainings had a longer-term effect, clients’ knowledge was reassessed through a subsequent survey, conducted approximately 6-8 weeks after the training.

As a guideline, the following process evaluation indicators should be kept in mind:

• Targeted – Indicators must measure what is changing (elements of change), who is involved (target group), where the change took place (place), and when the change will/has happened (timeframe).
• Measureable – Indicator must measure units that have changed (units of measurement), capture the measurement at beginning (baseline measurement), and capture the quality of the change (whether change is effective, appropriate etc).
• Reliable – Indicator must be credible, assumptions about its usage should minimal, and connections between indicator and what is being proven should be clear.
• Feasible – Using the indicator must be feasible and viable. Measurement must be doable so that information can be obtained.

Through a process evaluation, program managers can better understand and evaluate issues arising during implementation of a process – operational (includes service delivery, methodology used and resource usage) or programmatic (includes service usage and participant experiences). This evaluation can be conducted during various phases of the project – mid-term, upon completion and after completion to ensure the issues surrounding the implementation of critical processes are well understood.

Detailed results from IFMR's evaluation are currently under review; however, early results indicate that we can further improve program implementation based on client and trainer feedback. For example, some trainers and the clients wanted more regional terms to be used during the financial literacy training. This was despite that IFMR spent nearly six months developing materials and conducting pilots in rural areas of Bihar and Uttar Pradesh prior to program implementation. 

We will be sharing detailed results on the imact of financial literacy on branchless banking clients soon.

--------The author is senior manager, Policy Impact and Development, IFMR-Research, India