Session 4: Matching quantitative methods to uses

October 30, 2012 1:00 PM to 5:00 PM EDT

Different methods are useful for answering different questions. Clarifying your research question is the first step to gaining insights that are useful from a business standpoint. This session will leave participants with a better understanding of the questions that each quantitative tool is best suited to answer.

This conference has ended. Please feel free to peruse the commentary for each of the 10 sessions.

Presenters: 
Kabir Kumar
Microfinance Specialist

Kabir Kumar helped launch CGAP's program on technology-enabled business models for financial services and has worked closely with some of the pioneering implementations in the mobile financial services space, including Easypaisa in Pakistan and Eko in India. He manages CGAP's key relations with businesses, especially in the global mobile and finance technology space. He has been an adviser to banks, mobile network operators, technology companies, and investors in over 15 countries in Asia, Africa, and Latin America. Kumar has led the Technology and Business Model Innovation team's work on business case questions where he has identified alternative models. He continues to identify new areas of business opportunity, such as the role of data and digital footprints for financial inclusion.

Claudia McKay
Financial Sector Specialist, CGAP

Claudia McKay works with the Technolgoy and Business Model Innovation Team, focusing on research areas related to the large-scale adoption and usage of branchless banking by low-income, unbanked people as well as the business case for agents. Prior to working at CGAP, she spent seven years working for Opportunity International, a global network of microfinance organizations providing more than 1 million of the world’s poor with financial services. She worked as the director of product development for the global network, developing innovative financial products (such as agriculture microfinance, housing loans and index-based weather insurance) that meet the diverse needs of Opportunity’s clients. She also spent four years working as the head of microfinance banking at Opportunity Bank in Malawi, a commercial microfinance bank.

Jake Kendall
Bill and Melinda Gates Foundation

Comments

Submitted by Claudia McKay on

Welcome to Session 4! In this session, we will be discussing matching quantitative research methods to uses. As we’ve discussed in the previous session, quantitative methods can be a powerful tool to gain customer insights. Quantitative (as opposed to qualitative methods) are particularly useful to quantify a problem or potential market size and understand how prevalent it is by looking for projectable results to a larger population.

There is a wide range of quantitative methods, from small, inexpensive surveys of customers as they leave a branchless banking agent, for example, to large-scale randomized control trials that seek to measure the welfare impact of financial inclusion across an entire population. How do you know which method to use? If you match the wrong method for your particular research need, you might end up wasting a lot of time and money and may end up needing to start over with a different research method. So, it is vital that you first clearly define your research objective and develop a working hypothesis. Once you know exactly which question you are trying to answer, you can assess the various methods to determine which is best.

We can begin our session by discussing some of the most common questions or uses of research and talk about which research methods might be best for each use and why. Some common objectives are:
a) Market level data that can be used by a wide set of actors (e.g., Finscope surveys or other surveys measuring perceptions on financial services, issues and access in a country)
b) Market sizing or feasibility study for a provider when deciding whether and how to enter a particular market
c) Customer segmentation to decide who to target and how
d) Product development to understand which dimensions of a product are most important to customers and how customers react to changes in product features
e) Feedback from customers on satisfaction of a particular product or service, especially when there are low usage or retention rates
f) Welfare impact to understand if the product or service is improving people’s lives

As an initial question for discussion, let’s start with market level data. Which quantitative methods are useful for understanding data across an entire market? Have you had any experiences where the ‘wrong’ method was used to do this and the results were not very useful?

Session 4 Moderators: Claudia, Kabir and Jake

cgap

Submitted by Steve Wright on

Poverty measurement data is a critical component if an explicit goal of the MFI is to serve the poor. This data is most useful at a granular level where it can give an MFI insight in to their targeting/outreach of the the poor, there product performance by poverty line, and in a perfect world, the progress out of poverty of their clients. The primary roadblock to this being easy and efficient is Management Information System (MIS). In order to correlate household level poverty measurement data to other operational variables (gender, product consumption, etc) an MFI needs a strong MIS with well managed unique ID's and at least some basic reporting and analytics capacity. This, in my mind, is "research" from the perspective of an MFI. And, it is a clear and well defined pain point. Specifically, the ability to take the data they have and make sense of it in a way that can inform business decisions.

Grameen Foundation

Submitted by Tanaya Kilara on

Recently the Global Findex data released by the World Bank has been very useful in terms of market level data and facilitating cross-country comparisons (http://datatopics.worldbank.org/financialinclusion/). But are financial service providers using this data or is this more useful at the policy level? I would love to hear the experience of providers in using market level data for business decisions.

CGAP

Submitted by Michel Hanouch on

Thanks Tanaya. I had a similar question relating to the use of FinScope type data. Does anyone have concrete experience of providers using this data extensively, and if so, for what specific use? If not, do we have a sense of what can be done to encourage providers to benefit from these types of studies? Is the format that the data is presented in a key constraint?

CGAP

Submitted by Leora Klapper on

Thanks for bringing up the Findex data, which is the first comparable data on the use of savings, credit, formal payments, and insurance by adults in 148 countries around the world. The data can be accessed directly at: www.worldbank.org/globalfindex. We've been seeing reports from industry participants (MasterCard, Citi, etc.) that use the data. However, we hope it will be used more broadly to identify market opportunities--for example, we identify a large percentage of adults (in the FSU, MENA, etc.) that have formal accounts but use ONLY informal credit and savings methods. Data on these so called ‘underbanked’ can be used to motivate and design new financial products that meet these clients’ needs. Same for the large percentage of adults (particularly women) in Sub-Saharan Africa that commit to regular savings at ROSCA’s and other community savings clubs. Our website offers various tools to download, map, and graph the data—but suggestions on ways to make it more user friendly are always welcome!

World Bank, DEC

Submitted by Kabir on

Steve, how does "poverty measurement data" compare with other tools to collect market level data like the Findex data that Tanaya just brought up?

CGAP

Submitted by Narasimhan Srin... on

I tried using it in Tanzania for looking doing some demand estimates for rural finance, but found it difficult. When a large part of the demand comes from uncharted territory (people who are not near the financial sector, and a reliable for sampling among such customers did not exist, the representativeness of the data was a question in my mind. Information coming from surveys in local areas and some of the smaller financial institutions closer to the ground did not match the larger conclusions - but this kind of mismatch is to be expected. After some examination, I found that I was not comfortable to make use of the data.

consultant

Submitted by Steve Wright on

So I am not that familiar with the Global Findex data but in general terms, Findex and other world bank data sets are incredibly valuable "background" data sets. I am not sure that I would classify Findex as a "tool to collect market level data" and maybe I am just splitting hairs on the verb "collect". I can go to the world bank and get regional instance of poverty and rates on financial inclusion and then I can measure the poverty of my own client base and compare the two. If the regional instance of poverty is 80% and 30% of my clients are poor, then I have some work to do. Same with financial inclusion from Findex which can help me understand gender or poverty relative to my financial services products.

So, when I say "poverty measurement data" I mean data that an MFI has gone in to the field to collect from that MFI's clients. The MFI needs "tools" to make that process efficient and effective. These tools are MIS and mobile phones and reporting and analytics tools.

Grameen Foundation

Submitted by Kabir on

Has anyone used Finscope?
I am wondering if there are folks out there who are familiar with Finscope and have used it and could comment on how it compares with other options available to providers on market level data?

CGAP

Submitted by Mark Napier on

Most definitely it has been used - in fact, a few years back we (at FinMark) did an impact assessment of FinScope precisely to try to get a better fix on who was using the data and how it was used. I am sure that FinMark would be willing to share this with you. Interestingly, one of the conclusions was that financial institutions with more sophisticated marketing departments were really important targets for FinScope because, once the FinScope segmentations were integrated with the segmentation approaches used by the FIs in question, they never wanted to turn their back on FinScope. Smaller institutions do need more support anlaysing the data and turning data insights into actionable product development or marketing strategies (it is one of the things that, in my new capacity at FSD Africa, I would like to work on) but there are many, many examples of FIs (of all shapes and sizes) who have used even the headline data to refine market entry strategies and justify investment in product development. Big bank users include Absa, Barclays (in Zambia, for example), Standard - the problem has often been that FinMark did not always know when the data was being used (no need for the FIs to tell us!). Only today I was reading EFInA's 2012 Impact Evaluation which contains the following excerpt: "The new product development team at Diamond Bank had the following to say: “I do not know where banks would be without them (EFInA)… our entire retail team uses their information”". So, yes, unequivocally - it is used (....but not as much as it could be). And the FSD network would be able to give you more detail on just how.
As regards alternatives, again, the impact assessment in FinScope I referred to has something on this (if I recall correctly). Clearly, value for money is a big attraction (either free, in most countries, because donor funded, or very cheap, where the costs are syndicated across a number of users (as in SA). The explicit focus on emerging consumer segments (ie questions designed with this purpose in mind) is also a real attraction. The analytical back-up available from FinMark was also cited positively. As to whether competitive offerings have come into the market in the past few years, I am not sure.

FSD AFRICA

Submitted by Nara Hari Dhakal on

I will be grateful if any one explain me briefly about Finscope? I am interested to learn more about it and explore the possibilities of using it in Nepalese microfinance sector.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Shiv Kumar on

Main source of income for Rural poor is Agriculture. Surveys which helps in analyzing the historical information on Crop output, Land fertility & climatic condition plays a major role.
Most of the surveys are mainly focused on the income level, factors which effects the income levels should also be part of surveys.

Hugo Technologies

Submitted by Kabir on

Which type of data matters most?
Steve, Shiv, Tanaya, Michel and others: Providers often want quantitative data over qualitative data. I am not sure if this was discussed in the previous session, but I wonder if we can take a step back to answer -- what is the best role for quantitative data on customers? Is quantitative data superior as many providers claim and believe?

CGAP

Submitted by Tanaya Kilara on

Kabir, I think quantitative data gives providers a level of comfort in being able show their Board concrete numbers and percentages. Whether quantitative data gives deep customer insights is still an open question for me.

CGAP

Submitted by Kabir on

Perfect marriage of quantitative and qualitative?
Has anyone seen a provider do well with balancing consumer research that is both quantitative and qualitative?

CGAP

Submitted by Shiv Kumar on

Cannot comment on which is superior. I will leave the judgement to the specialist.
However, Availability of quality of seeds, Fertilizers, Rainfall measurement, Land fertility are some of the qualitative measurements which has direct impact on income levels.

Hugo Technologies

Submitted by Steve Wright on

For me, qualitative data is informed by quantitative data. Qualitative data is narrative and relies on small "sample size" so quantitative data is necessary so that you can "click through" the qualitative to get to some supporting detail.

Grameen Foundation

Submitted by Nara Hari Dhakal on

Determining relevant data, based on research questions/hypothesis to be tested, is important consideration for undertaking data need assessment. Only properly assessed data need can provide basis for determining the types of data that matters most in quantitative research.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Edward Cable on

Echoing the comments of Steve on the importance of being able to track and analyze this poverty measurement data from directly within the MIS, we have directly integrated tools like the PPI into our Mifos software. Some of our largest users like Grameen Koota have been able to incorporate reporting on these quantitative social metrics into their management decisions because of the integration between the MIS and the data collection. This was well-documented in a recent case study that Grameen Foundation had commissioned: http://progressoutofpoverty.org/new-case-study-grameen-koota-tracking-cl...

We want our software to go beyond tracking of specific poverty measurement surveys and have built in the flexibility to capture any form of client data that the MFI would like to track that can be uniquely identified to the client and reported on accordingly. Part of the forward-looking vision of our software is having a data model and an architecture that supports a client-centric view. Technology should facilitate operations in which the client is the focal point and not merely the financial service they're being provided. This is manifest in the ability to capture and monitor whatever data they so choose, easily integrate with local credit bureaus, seamlessly generate and store client exit surveys, capture surveys like the PPI both online and through the mobile phone, etc.

Some of our users have very extensive processes before they accept clients into their financial services programs - we want to support this holistic engagement by tracking the full lifecycle of a potential client and all the corresponding factors in the MIS such that it is not merely a portfolio management system but also supports the same data you would capture in a CRM.

The Community for Open Source Microfinance (Mifos)

Submitted by Kabir on

Should MFIs/providers do this kind research on their own?
Steve and Edward: do you think MFIs should be doing this kind of poverty measurement on their own, even if they have the MIS? I am wondering what other challenges you have encountered. Or it is better done as a public good at the market level?

CGAP

Submitted by Edward Cable on

I think MFIs should be doing poverty measurement on their own but as technology provider we should facilitate the ease of connecting to these larger sources of market-level data.

I think we are only scratching the surface of what can be done if we both make it easier to bring in analysis of larger sources of data and make it easier to make their data available for widespread analysis. I think it must be done at a market level but from the experiences of our users this is only now happening for very financial-oriented data like credit bureaus.

We have users that are capturing separately outside of their MIS extensive metrics like the MPAT (see http://www.nuruinternational.org/blog/category/monitoring-and-evaluation/) but we would like to help the MIS enable one centralized view of the client and the corresponding poverty metrics across their various programs.

The Community for Open Source Microfinance (Mifos)

Submitted by NIna Holle on

I am wondering if it is actually possible to establish a better link between academic researcher and MFIs, DFIs, etc. who want to conduct this kind of research. I know that microfinance has been becoming a popular field of academic research in the last years (and I am not only talking about MIT, etc. but all kinds of universities internationally). And we can find all kinds of research approaches here - quantitative, qualitative, even new experimental approaches. Can't we make better use of this? Has anyone experience with reaching out to researchers? Or the other way round: are there researcher participating who can tell more about their experience in cooperating with MFIs? Is it even feasible to conduct academic research on a large scale in the field?

CGAP

Submitted by Steve Wright on

In general I do not think it is the role of an MFI to do research. That said, "research" is a very generic term. MFI's who have as their mission to alleviate poverty must measure poverty if they are to know to what extent that are addressing poverty. As Edward states, we need to build enabling technology to make this as easy as possible. Again, the purpose of this data is so tht the MFI can get a very clear picture of their customers so that they can serve them better. This is completely analogous to Nike wanting a better understanding of people who buy tennis shoes.

Grameen Foundation

Submitted by Nara Hari Dhakal on

This type of research are in different forms. Loan officers, branch managers, head of branch management department, etc. who are concerned on identifying clients hardly consider issues of sampling and sample size determination. Their research is informal, not well structured and MFIs need to conduct such research on their own, while other types of research generating information on policy significance need to be conducted by independent research institutions or researchers to ensure validity, unbiassness, and reliability on research findings.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Alexia on

Kabir and others,

I think it is a bit of a false debate to position the question as whether quantitative data or qualitative data is superior. Going back to Claudia's opening post, I think any serious provider needs both for different reasons at a different times. If a provider is not yet working with low income people and wants to "size" the market, they will want to some hard numbers around number of potential new clients and the volume of business they might generate. But qualitative data is key to understand the needs, preferences, and behaviors is this potential new market segment.

In the same vein, I really do think that market level tools like Finscope and Findex are often the appetizer that whets the appetite of providers....the questions the data raises can help shape the research the provider will do more directly -- and often mining their own data (if as Steve ays the MIS is robust enough) is one natural next step.

CGAP

Submitted by Kabir on

Not a false debate for providers
Alexia, you are clear about the choice but it would be interesting to hear from providers who often are not so clear. It is especially not a false debate for providers with limited resources, in my experience.

CGAP

Submitted by Kabir on

Use the data you have?
In our opening post, we list a number of quantitative approaches/type of studies. We have been discussing market level studies. I am wondering what you think of providers using the data they already have instead of going out to the market for more quantitative information? Especially lenders, don't they arguably already have the best quantitative data on their borrowers?

CGAP

Submitted by Diana Lewin on

I have participated on a product development process which relied both on quantitative and qualitative data in a synergistic manner. The quantitative data served to quantify existing clients' use of a bank's services against the competition and to identify some of the barriers to usage. However, only after conducting the qualitative research, the bank was able to understand why people were not using the accounts and where the customers were actually saving their money and WHY. One of the most interesting lessons from this experience was the importance of each word used in quantitative surveys and the different ways in which it can be interpreted. On the quantitative survey, the majority of customers agreed that the bank was a safe place to save. However, the qualitative research revealed that although customers understood the value of saving in a formal institution, they didn't consider the bank such a safe place to save, as their balances would decline over time due to high costs of maintenance and transaction of the accounts.

MicroSave

Submitted by Claudia on

Hi Diana, thanks for the comment and for bringing up product development. I think that one of the major purposes behind research is often to develop a new product or improve an existing product. What quantitative methods have others used in product development and what has been their experience? Are there any examples where quantitative methods have yielded great customer insights for product development or can quant. research mainly size a certain market and help providers know where to target (but not necessarily with what product)?

CGAP

Submitted by Alexia on

Who design the surveys, who interprets the results?

I also have a follow-up question for Diana. You post underscored that the INTERPRETATION of survey results are key. And some qualitative design arguably provide more back and forth for testing that the survey results are being understood how they were intended. I'd like to do whether the people designing the surveys are the ones that will ultimately use them. Let's take the product development example -- is that a tight connection between the surveying team and those with the skills and authority to design new products?

CGAP

Submitted by Diana Lewin on

Dear Alexia and everyone,
The research I was sharing earlier was conducted by well-trained staff from many departments of a commercial bank. I was coordinating the research activities, data consolidation, interpretation of results and product design process. In qualitative research, we generally use open-ended questions and then listen to clients' responses. Therefore, the interpretation process is different than in a quantitative survey, as the client's point of view is being understood during the interaction.

With regards to your question about the connection between the survey team and those with the skills and authority to design new products, having the bank staff as part of the research and product design team was a key success factor in the product development process for many reasons:
1. We were working with a commercial retail bank, where most of the employees lacked knowledge of the segment. Having some of the staff interact with the target customers served us later to promote the importance of the project and get buy-in from many departments.
2. At the same time that the multi-disciplinary team was listening to clients' insights, they were analyzing them in contrast with the bank's strengths and weaknesses and thus were able to understand the relevance of overcoming certain issues as well as devise solutions that were appropriate to the bank.

MicroSave

Submitted by rod dubitsky on

This may be a naive question but aren't we conflating two issues: 1) quantitative v qualitative DATA and 2) quantitative v qualitative ANALYSIS. You can have purely qualitative data such as life story narratives but if collected across a broad enough sample the qualitative narrative may be convertible into data elements where more advanced statistical techniques can be applied. Conversely you can have purely quantitative data that can analyzed in a qualitative manner by simple parsing and charting of the data and obviously you can apply more advanced statistical techniques. I think the range of qualitative and quantitate analysis and data is huge and bridging the camp between the pure RCT disciples v. those who prefer qualitative methods and analysis could create a range of analytical standards that would allow more consistent analysis across various studies. Frankly when I read RCT studies across seemingly similar studies have different outcomes simply because of the choices made by the analyst. Likewise, sometimes some studies findings are rejected due to insufficient analytical rigor.

BRAC

Submitted by Michel Hanouch on

Thanks Rod. Coming back to the initial post, do you have a good sense of which uses would benefit more from the quantitative versus qualitative data, and which from a combination of qualitative and quantitative data and analysis?

CGAP

Submitted by rod dubitsky on

I was thinking more along the lines of triangulating qualitative approaches with quantitive approaches. So for example, in estimating the impact of microfinance, if an RCT study concluded that the impact of microfinance had no impact, and that conclusion was separately validated by conducting random surveys that collect detailed survey responses then that would strengthen conclusion of RCT. Alternatively, if at the same time an RCT was being conducted, detailed diaries were kept, one could review the diaries and summarize and conclude qualitatively what the impact was and then compare to RCT results. What if survey universally recorded a net positive impact, at the same time the RCTs showed no impact? One could then review RCT design (was treatment and control adequately executed) and evaluate what might the diaries be missing that could exaggerate the impact. Triangulating between the two approaches may yield a better answer than relying on one or the other.

And regarding of uses, I pose the question - does product design (eg moving from group to individual lending or from weekly to monthly payments) lean towards a more qualitative approach while impact assessment can tilt more quantitative? I pose that as a question, as I'm not sure I have a good answer.

brac

Submitted by Claudia on

Edward, Steve and Kabir have raised the point about poverty measurement using the MIS systems of MFIs. MFIs and other financial service providers are often sitting on a goldmine of data that they barely analyze, and yet they then spend a lot of money getting research firms to undertake large surveys. How can providers really use and analyze their own data first before hiring the research firms? Are there any examples of good customer insights arising from data mining of the data already on hand?

CGAP

Submitted by Steve Wright on

This is an essential question that we must answer.

Grameen Foundation

Submitted by Alexia on

Excellent point, Rod. Can you give us first-hand experience from BRAC?

CGAP

Submitted by rod dubitsky on

Tough question. I can probably answer best by referring you to their website http://www.bracresearch.org/ (for those not already familiar). BRAC has always published significant amount of quantitative research, but they also embrace highly qualitative approaches. Within the quantitative approaches they have used propensity score matching as an alternative approach to RCTs.

brac

Submitted by Monique Cohen on

I would like to introduce a different technique, transactions analysis, that uses a quant and qual approach to product change. Using data from financial dairies and MIS we track individuals and household’s economic and financial transactions over time. The data provide insights into how people use money in real time. Dimensions of the analysis focus on the sequencing of transactions, the social networks through which transaction occur and the spatial dimensions of these flows. This data has been used in Malawi not only to identify new market opportunities but also to facilitate the management of cash flow, to improve risk management and to encourage asset building. Using this mode of analysis we have lso been able to understand to understand how dormancy occurs and readjust products accordingly. In a world in which ‘cash is king’ this mode of analysis can identify new opportunities for mobile money which might be unviable for bricks and mortar banking. The advantage of this approach also lies in dealing with new indicators of viability, transaction data.

Microfinance Opportunities

Submitted by Narasimhan Srin... on

very interesting

consultant

Submitted by Claudia on

Hi Rod, I think you raise an interesting point. I think that most often quantitative and qualitative research are very distinct, in part because researchers tend to be of one stripe or another. However, as Diana and Alexia point out, the interpretation of results is key and perhaps it would be helpful to have more interplay between quant and qual methods in order to encourage the best outcome. I think that a few rounds of focus groups or face-to-face interviews are very important to figure out the right questions and the right phrasing to put into a wider scale survey. In the product development example, perhaps quantitative methods can help size the market or conduct customer segmentation but then qualitative methods could help to really get customer insights. We'll be discussing this more in tomorrow's Session 8.

What about really understanding which aspects of products are most important to customers? Has anyone used RCTs to do this (e.g., offered different interest rates or loan terms to different groups to see how much impact this has)? What are other methods to understand this?

CGAP

Submitted by Nathanael Goldberg on

Hi Claudia,

Agreed on need for both quantitative and qualitative research. I would stress that well-designed randomized trials do get at client preferences and mechanisms for impact by isolating specific product components and what happens when you adjust them. We did a review of many of these studies with CGAP, available here: http://www.cgap.org/publications/latest-findings-randomized-evaluations-...

You can do focus groups or client interviews about what clients think about specific aspects of products, and those can be really helpful when designing products, but ultimately it's nice to be able to offer different flavors of the product in the real world and measure exactly what the client response to those changes is. What happens when you remove group liability, what happens when you give clients lower prices or a longer grace period, or offer them insurance? Or really specific: what is the take-up for insurance when a representative from their trusted MFI comes along to market it? Answer: 10% increase in take-up. Not easy to answer without a field test.

Nathanael

Innovations for Poverty Action

Submitted by Narasimhan Srin... on

If the quantitative and qualitative analysis (I am not judging the superiority of any) actually provide quality decision support, how does the financial sector which uses these analytical frameworks extensively have products that are unsuited to customers, that bleed, that are difficult to market? Especially in insurance, with so much of data based analytics going on, customers get a lot of fine print and one way policies that take in premia and deny claims in case vulnerable people? Is it a problem with our analytical tools that prevent good designs, or is it weak interpretations that mislead or is that the suppliers assumptions are speciously validated by these techniques?

consultant

Submitted by Elisabeth Rhyne on

I would like to go back to the Findex data. I have learned a lot from looking at the Findex data, even thought it is at present only available at the country level. Next month, however, they will release the detailed data, which will have observations from 150,000 individuals!
Findex data is so important because a) as demand-side information, it cuts through lots of shortcomings of supply side information that has been more readily available; and b) as a cross-section of the population, it covers non-clients as well as clients.
One way to use Findex data is to develop hypotheses about client behavior that can be tested in more depth in a given situation. For example, Findex data shows that while 41% of adults in developing countries have bank accounts, only 7% of adults have bank accounts that they use actively (defined as making more than 2 withdrawals per month). There's a mystery that any organization seeking to offer savings services in these markets would want to solve!

Center for Financial Inclusion

Submitted by Narasimhan Srin... on

150000 seems to be a large sample but not really so in many contexts. In India a survey of household debts and investments is carried out every 10 years which has a sample of about 100000- the data is extensively quoted and used but at the end of the day all of us who use it do so with reservations because it is such a tiny sample for the country the size of India. Specific field studies that are carried out in some locations do not support the survey findings fully. But for want of any other similar information, this survey has been taken as the basis by almost all planners, academics and researchers in their work realting to access to credit- and it has been difficult to make adjustments to the data as we do not know what has to be adjusted . The use of different and large number of surveyors in different language areas make the quality of data uneven. There are interpretation and understanding issues in the hands of different surveyors and this impacts of the final outputs. But then it is an imperfect world and we have to make do with sub-optimal quantity and quality of data - the point is that let us be conscious of it even as as we take far-reaching decisions (some of these seem to be daring leaps of faith given the underlying information base).

consultant

Submitted by Nara Hari Dhakal on

I think choices of technique: sampling versus census is the first very important factors to be considered to match quantitative method to uses. If census is to be used, sample size is irrelevant while if sampling is to be used sample determination deserves careful consideration.

Sample size can't be said or determined flatly on adhoc basis. It's determination should be based on population mean, population standard deviation and degree of precision (confidence/significant level) the researcher seek to establish. So statistically we can't say this much sample is representative or adequate, this needs to be determined scientifically and there are statistical methods to determine it.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Elisabeth Rhyne on

I would also like to highlight the value of data that is not directly related to financial services. As part of CFI's Financial Inclusion 2020 project, I've been involved in looking at demographic trends and have found that they generate a lot of new insights. For example, demographic trends in middle income countries (BRICs and similar) shows that mature families and older adults are an increasingly important demographic segment. Exploring their financial needs as a distinct market segment could be very fruitful for providers and policy makers. Same goes for looking at economic data and many other types of information.
We will be releasing a paper on the demographic analysis soon, and CGAP is also working on this issue.
Data at this big picture level is mostly useful for formulating hypotheses to be investigated in specific markets.

Center for Financial Inclusion

Submitted by Jake Kendall on

Interesting point Beth. I believe the Findex data shows that many people use bank accounts just to recieve a wage or government transfer, then withdraw. So part of the question may be, how do we convert those people to clients who use the account for more use cases?

Bill & Melinda Gates Foundation

Submitted by Monique Cohen on

Much of the market research we do begins with an assumption that the consumer desires a certain product. S/he buy a house with cash. We posit the research around the question about how could formal financial services facilitate this purchase. Nice idea and well intentioned except from the perspective a consumer a loan to pay for their most valuable assets is not optimal. They prefer a home, paid in cash to buy now/pay later. Who know what can happen in between? Traditionally we have designed housing microfinance loans by focussing on their necessary attributes. However, maybe another approach is to structure a product around cash flow management practices and the recognized priority to build an asset. Would the product look different, who knows? However, what can surmise from experience is that the new will likely work better because it fits the money management practices of the borrower.for more information please go to www.library/putting-clients-center-designing-and-delivering-effective-fi....

Microfinance Opportunities

Submitted by Jake Kendall on

Beth, I like both of your posts as they highlight a different use of research and data than I often see out there. I think it contrasts with some of the approaches highlighted earlier.

IRoughly there seem to be two approaches to data and research. The first is to scope data collection or reserach then try to drive toward recommendations for specific product or service features that meet the client needs revealed in the data. Some of the posts above refere to the need to motivate research in this way. This usually implies a particular approach involving looking at how people use the informal products they already have or asking them how they might use a new product with a given set of features (e.g. a group micro loan).

The second approach (which I think you are taking here) is to study financial behavoirs more broadly, or demographic trends, or other economic phenomenon within the household (e.g. the prevalence of shocks) where there is not necessarily an obvious resulting recommendation for product development but where the information could inform hypotheses development and eventual product approach.

Does you (or anyone else) have view on which approach is more valuable from the provider perspecitve? Any good examples of either type of approach you like?

I ask because as a funder, I wonder what the right mix is in the research we fund.

Bill & Melinda Gates Foundation

Submitted by Monique Cohen on

Much of the market research we do begins with an observation that s/he buys a house with cash. We posit the research around the question about how could formal financial services facilitate this purchase. Nice idea and well intentioned except from the perspective of a consumer a loan to pay for their most valuable assets is not optimal. They prefer a home, paid in cash to buy now/pay later. Who know what can happen in between? Traditionally we have designed housing microfinance loans by focusing on their necessary attributes. However, maybe another approach is to do the research around cash flow management practices and the recognized priority to build an asset. Would the product look different, who knows? However, one can surmise from experience is that the new will likely work better because it fits the money management practices of the borrower.

Microfinance Opportunities

Submitted by Nara Hari Dhakal on

Yes I agree. Collection of background data is very important to properly match quantitative methods with uses. In order to define a sampling frame, population must be defined. Considering the size of the population, choices of sampling techniques, confidence level and degree of accuracy, sample size should be determined.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Alexia on

Is all the above not a realistic answer?

Again, Jake, I find myself saying that this is not an either or choice. This said, I think we can distinguish the kind of "public good" research where one investment can be shared and used by many. I do not think it would be a good use of providers' resources for each and every single one of them to research demographic trends. Yet, a client-centric provider (for frankly even simply a provider with good business sense) will what to understand these trends. I remember a provider that is part of the Youthsave Consortium looking at the average age of clients in their bank, looking at the average age of the population and thinking that if they did not act today, they simply would have no clients in the future. Some of the more product specific research, however, a provider might have to do themselves to then be able to truly act on the new insights.

I also want to go back to Monique's fascinating transactions analysis that uses both quant data from MIS and qual data from the diaries. Monique -- is there a paper you can share. I like that the focus appears not be on product per say, but on meeting people's needs and behavoirs (though I know Monique does not like to speak of people's needs). But I am referring to risk management, cash management, and asset accumulation needs.

CGAP

Submitted by Marten Leijon on

I would agree that there is a need for all --and opportunities to integrate insights.

Whereas efforts to drive towards conclusions for specific product or service may generate more precise findings (certainly to the question for which they were designed), approaches that aim to create a more holistic view of the user's needs and behaviors in a more inductive way help make sure providers (or funders) are asking the right questions.

Perhaps I am simplifying things, but I view the former as research to support strategy for the evolution of products (or channels), the latter as research to support strategy writ large. Clearly, a thoughtful collection of product/channel focused inquiries can build a pretty complete view of the client, but the (perhaps more inductive) approach of looking for broader patterns can be more powerful in driving true innovation.

In terms of techniques for this latter approach, I would take a closer look at the approaches to building customer personas as a way to capture deeper insights of the more holistic kind, integrating quant, qual and diary studies.

MIX

Submitted by Jake Kendall on

I think that is a good distincition, what is done for public consuption may by necessity be more general, while providers will of course focus on their specific products and client populations.

Bill & Melinda Gates Foundation

Submitted by Jake Kendall on

As you know Monique, I really like this line of work you guys did. I actaully think it bridges the distinction I was trying to make earlier as it both helps understand very general and fundamental financial phenomenon and processes in poor households and you also drive to some product recommendations in the reports.

Bill & Melinda Gates Foundation

Submitted by Elisabeth Rhyne on

It's important for people carrying out market research to have an understanding of the financial lives and indeed the lives in general of the prospective clients, and one of the important functions of market research is to test whether that hypothesis holds true and to refine it in a specific context. I don't think it's a question of one kind of research vs. the other, Jake, as you suggest, but a question of the interplay between the two.
We criticize political leaders for narrowly following polling data and forgetting to lead. I think there is a parallel here -- conducting market research without a deep understanding of the lives of the clients (of course open to change with new evidence) can lead to tweaking products that aren't the right fit for clients in the first place. Something like that can be seen in the microfinance world's over-emphasis on credit because of a faulty hypothesis about client lives. Well, maybe it's a partial parallel....

Center for Financial Inclusion

Submitted by Jake Kendall on

Agreed - and well put! I wasn't necessarily proposing we choose between the two approaches, was more hoping to get some debate on when each is appropriate and what the right mix is.

In that vein, here's another question. When research is aimed at more general understanding of the lives of clients rather than focused on specifics to drive product decisions, its easy for it to lose relevance. What are the areas we should be researching in the more general category of research? What phenomenon are not well enough understood? What data would help drive innovation (as Martin suggests is one thing that research looking at broader patterns can do)?

Bill & Melinda Gates Foundation

Submitted by Claudia on

I think that several of the above comments are all leading to a similar conclusion. Narasimhan wondered why with so much research going on, financial products for the poor are still largely low-quality and not meeting their needs. Monique gave an interesting example of purchasing a house that shows that those who design products start with an end product in mind instead of truly understanding the cash flow needs of their clients. And Elisabeth rightly points out that doing market research without a deep understanding of the lives of clients again does not lead to quality offerings. Clearly, the best research in the world is useless if it is not a) designed and carried out in a high quality manner, b) interpreted well but also c) applied and translated to real products and services in an effective way. I think that providers who understand something about research and are involved in the research process from beginning to end have a higher chance of actually applying the research in the right way. What have the experiences of others been in terms of outsourcing research completely, doing it in-house or some sort of a hybrid to get the expertise of researchers but keep it tied to business needs?

CGAP

Submitted by Tanaya Kilara on

Claudia, in my previous life working at a start-up microfinance institution, we had to bring in outside expertise to conduct market research for new products we were designing. The challenge was to integrate the research into the eventual design of products. It would have been great to have in-house expertise, but that is a challenge for smaller, resource-constrained organizations. Also, I have often wondered about whether financial institutions would have enough 'business' in a sense to justify an in-house team and whether that team has the incentives to stay current and cutting-edge. These are questions I grapple with and don't know if there is a silver bullet. I am very interested in hearing from other organizations, especially about the hybrid model you talk about. The Microfinance Gateway also carried a series raising some of these questions last year - http://bit.ly/ygcB9P

CGAP

Submitted by Claudia on

To Jake's point, some important pieces of research in recent years (e.g., the financial diaries that led to Portfolios of the Poor) were aimed at more general understanding of the lives of the poor. I don't know if there is one specific product or service that arose from Portfolios (although there very well may have been) but it certainly has influenced the entire community and motivated many providers to look deeper at client insights and to understand the full complexity of financial needs beyond working capital loans. What else should we be researching? I still think we don't understand informal products and the very valuable role they play in poor people's lives. We tend to broadly stigmatize them for being unreliable and expensive yet poor people continue to use informal products even when they have access to formal products. What kind of research can help us better understand what specifically could be the value to poor people in switching from informal to formal products and how we can improve that value proposition? More financial diaries? In-depth interviews? Surveys?

CGAP

Submitted by Claudia on

Thanks everyone for your insightful contributions to this session. In the past four hours, we have grappled with many topics under the general umbrella of matching quantitative research methods to uses. We've discussed how quantitative research is used to create public good data like Findex and talked about the limitations of the best quantitative research methods when they are not interpreted and applied properly. We also touched on how MFIs and other providers can use their own data for purposes such as poverty measurement. I think we foreshadowed a lot of tomorrow's session 8 on combining quantitative and qualitative methods and based on how often this came up, I think we will have a vibrant exchange tomorrow! For now, please stay around and participate in our final session of the day where we do a Wrap Up of the first full day of the conference.

CGAP

Submitted by Nara Hari Dhakal on

Sample size determination and sampling techniques are two important factors to be considered for matching quantitative methods to uses. Sample size could be small if pure random sampling could be used while it should larger enough if probability proportionate sampling or some types of randomization is used. Determining how far a particular sample size truly represents population under considerations is the matter of power analysis.

Further choice of statistical softwares (SPSS, e-views, strata, etc.), their availability and fluency of the researcher on types of relationship s/he seek to establish is important for matching qualitative method to usues.

Senior Advisor, Centre for Empowerment and Development, Nepal

Submitted by Nara Hari Dhakal on

Listening to and learning from the clients clearly requires understanding of research methods, which can be either qualitative or quantitative in nature. Quantitative research methods are descriptive and inferential in nature. There are different statistics and statistical tools used both in descriptive and inferential statistics. Choice of test statistics and statistical tools depends on key research questions (hypothesis) raised in the research. In other words, different methods could be useful for answering different questions.
In order to do the proper research, there is a need to clarify the research question. Such a research question should have significance from the business standpoint. In other word, there should be linkages of the research question on business and the answers (research findings) should have prospects of using for expanding outreach and scale of operation of the MFI.
Based on the research questions, statistical tools should be selected. It can be simple tabular analysis using frequency counts, mean, mode, median, standard deviation or it could be highly inferential using hypothesis testing tools (z-test, t-test, chi-square test, F-test, ANOVA), correlation analysis, regression analysis, etc. depending on type of relationship we are seeking to establish or nature of the hypothesis we are going to test. Thus, this is entirely a matter of the exposure of the researcher on statistics. In other word, application of statistics or econometrics on microfinance, i.e. listening to and learning from clients and type of relationship we seek to establish and use for business expansion.

Senior Advisor, Centre for Empowerment and Development, Nepal