Rich Rosenberg blogged yesterday about an Atlantic article that profiled John Ioannidis’s critique of medical research. The article reminded me of a meeting held in Washington a few years ago. Consumers and producers of microfinance impact studies were brought together to discuss the research agenda. One participant, who is not a researcher, concluded dolefully that microfinance research lags far behind medical research.
My immediate thought was that that claim was probably false: not because microfinance research is so far ahead, but because much medical research seems full of problems. If you read beyond the newspaper headlines, it’s usual to see simple correlations between health conditions and a given diet/activity/lifestyle quickly – and falsely — assumed to be causally determined. Sample sizes are small. Lots of hypotheses get tested, but just a few get published.
According to Ioannidis, it doesn’t get better when you scour the academic medical literature. The willingness to broadcast that fact has made John Ioannidis a rock star in the health statistics world. Health researchers routinely test many, many hypotheses; often rely on small samples; and face fierce competition to get published in top journals. One result, Ioannidis argues, is that the pressure to publish, combined with editors’ penchants for publishing striking (often counter-intuitive) results, means that a lot of results get published that wouldn’t hold up if the sample had been larger or the tests more robust. Ioannidis’s analysis holds especially strongly in non-experimental studies (his simulations suggest that 80% of “results” from non-randomized studies are in fact wrong). Another big result is that randomized controlled trials do much better (25% of health RCTs are wrong, according to Ioannidis).
Of course, being wrong just one-quarter of the time is no cause for celebration. But it does point to a real strength of RCTs – which is even more true of RCTs in microfinance .
The RCTs do better because they are designed from the ground up to test a particular (and narrow) set of hypotheses. That greatly curtails the opportunity for “fishing expeditions” and the chance that one of your 57 hypotheses happens to be statistically significant by a fluke. The questions tend to be far more focused in the microfinance context. Does access to microfinance increase business profit? Business investment? Household consumption? Those microfinance hypotheses usually stem from a clear theoretical model and should show up in clear patterns.
That’s far different from many medical studies, in which a much greater range of plausible hypotheses exist (along with a greater range of incorrect hypotheses). The situation persists because the specific pathways that link diet/activity/lifestyle to health conditions remain poorly understood. So lots of stuff gets tested in the medical literature, and “effects” may emerge that pass standard levels of statistical significance but which are caused by odd outliers or other features common to small data sets – and which turn out to be wrong.
So, on this score, Ioannidis’s criticisms are far less of a concern when it comes to microfinance. We have tighter theoretical understandings, a smaller set of hypotheses, and,usually, bigger samples.
But don’t relax completely. Microfinance research has its own set of concerns. Here are a few:
1) Replication. We don’t (and can’t) replicate studies in the sense that medical researchers can. Medical researchers replicate by trying a similar test again with a different, similar sample. It can be a big help in determining the robustness of the initial finding, over-riding results that do not hold when repeatedly replicated. But in microfinance, to the extent that we replicate, we do it to test the same idea in very different settings. Sure, it worked in Bosnia, but will it work in East Timor? Argentina? What we get is a mapping of the landscape, learning how financial mechanisms work in different contexts. That’s helpful to understand, but because contexts vary so greatly, a positive finding in one place rarely has power to knock out a negative result elsewhere. Replication is crucial, but not for the reasons that replication is crucial in the medical context.
2) External validity. The problem of replication above is tied to a more general problem in extrapolating from one context to another. This is hardly a problem unique to RCTs: it is a generic problem of evaluation. On this, researchers could do a better job of explaining who’s in their samples and how the populations relate to communities in other regions or countries.
3) But is it an interesting parameter? When there’s a debate about RCT results, it’s usually not about whether the finding is wrong or right, but about whether it’s interesting. Did the experimental design yield an estimate of an aspect of microfinance impact that matters most? Researchers deserve much credit for measuring the short-term impact of new urban microfinance branches, say, but if we had magical powers we’d really like to measure the impact of established branches in more typical settings, and we’d want to see longer-term impacts. It’s not fair to downplay the findings that we have just because we lust after an idealized (and unmeasurable) set of parameters. Still, we need to accept that a given study might not give us everything we want to know.
The problem with imperfect studies is that you don’t know how big the bias is (but you know the bias could be really big). RCTs have taken us a huge step forward, and promise to deliver clean estimates of various slices of microfinance impact. In the end, the big question is not the one Ioannidis asks (are the results apt to be right or wrong?). The big questions concern how the particular results matter to our understanding of microfinance broadly.
Dear Jonathan (and Richard),
What I would like to know is what your definition of Microfinance is, which I think is crucial if you want to do research on its impact. From the comments above I conclude that you define MF as loans for “economically active people” for poverty reduction. Is that correct? That would also put at the test of who “economically active people” are but that is another matter.
A few years ago CGAP defined MF as the building up of inclusive financial sectors. Testing the success of that activity seems to me to be quite straightforward; you collect and monitor how many different (verifiable) persons have a basic transaction (aka deposit) account (some FIs who are not allowed to collect public deposits have other similar products) with a micro finance institution (including banks with specific MF departments) and you follow how much money MF clients manage over time as assets. Following people’s outstanding loans has some drawbacks obviously.
As I am not an academic nor an economist I would very much appreciate your feedback as to whether the above assumption, following how much money people own in an MFI account over time, would be a valid and relevant test for the impact of Microfinance.
Thank you in advance and my best NY wishes, Peter
There was a fascinating article on a related topic by Jonah Lerner in the December 13, 2010 edition of the New Yorker, “The Truth Wears Off – Is there something wrong with the scientific method?” (http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer) The article deals with the peculiar phenomenon of the “decline effect:” it apparently happens frequently that the scale of the effect measured in an experiment declines when the experiment is replicated. The article extends the range of explanations for this offered by John Ioannidis.
Agreeing with Peter, I would like to further go deep into the issue related to definition of Microfinance as I consider vital for any research in MF arena irrespective of types of methodology and RCTs. According to CGAP, ‘Microfinance offers poor people access to basic financial services such as loans, savings, money transfer services and micro insurance. People living in poverty, like everyone else, need a diverse range of financial services to run their businesses, build assets, smooth consumption, and manage risks.
What I understand that MF is a package of financial services as referred to above which are all needed for the goal poverty reduction or poverty elimination. Logically there is a need to establish causal relationship between the MF services holistically and the poverty status in terms of income generation, empowerment, protected livelihood environ etc., for the given sample or the area . That is to probe extent of the availability and accessibility of all the MF services in a given sample or area and how have they influenced the level of poverty and the socio economic status of the sample or the area. If micro credit alone which per se not an adequate service for the said goal, is considered . probably it may hold good for the poor in top layer in the poverty pyramid or economically viable poor only (peter) . Even in such case, different socio economic cultural context influence the functioning of the financial mechanism particularly in the case of micro credit rendering difficulty for establishing direct casual relations.. Over a period of time micro credit has become be all and end all in MF arena. In such case a begging question ‘is it worthwhile to research on the impact on micro credit alone on the poverty’? Ironically how is micro credit alone is treated on par with the Micro finance concept in almost all the impact studies by the researchers/academicians ? Among the poor , different segments need different set of MF services and not necessarily micro credit alone. In the case of migrant labour, access to timely transfer services for their wage income from place of work to their native village certainly facilitate meeting their socio economic consumption needs and eventually poverty reduction . They also need micro insurance and safe saving places (on personal study among migratory labor who constitute significant proportion in poverty sector working construction industry ) and not micro credit on priority . In such case without much reliance on micro credit , there will be a good impact of MF other services referred to above on these segments – a good hypothesis to be tested in MF platform
What I wish to emphasize here that a clarity is needed by the researcher when they use the term Micro finance and usage of the term micro credit for representing micro finance is misleading . To me the term MFI is not correct to identify the institution which delivers only micro credit services unless all the MF services are arranged by them either singly or severally. Otherwise these institutions could be better called as Micro credit institution (MCI) distinctively . Obviously these institutions with micro credit input only could do little towards the mission goal of MF
Conceptual clarity and better appreciation on the mission are therefore needed in the research in Microfinance field