Latest Impact Research: Inching Towards Generalization

11 April 2012
3 comments

The most rapidly obsolescing part of my book, Due Diligence, is chapter 6, which reviews the statistical evidence of the impact of microfinance on poverty. Since I put the text to bed, working papers have appeared that test microcredit in Mongolia and Bosnia & Herzegovina and microsavings in Malawi and Chile (though the latter is marked “do not cite or circulate”). There’s also the Morocco microcredit study, which I didn’t catch wind of until too late in the book production. Add all these to the trials of microcredit in India and the Philippines and of microsavings in Kenya—the one that initiated this wave of research in 2009—and we have five credit studies and three savings ones.

A common exchange in the debates over the randomized studies goes like this:

“Microfinance has only been tested in a few times and places. You can’t generalize from that.” “True[I often reply]. But as it is tested in more places, if patterns emerge, it will become easier to generalize.”

I like to say that research is the pursuit of responsible generalization.

I’m excited that just in the last few months, it has become possible to generalize more responsibly about the impacts of microfinance. Fortunately, the latest results strengthen the conclusions of my book—they don’t show that microcredit cures poverty after all. More importantly, they enrich our understanding of the impacts of microfinance.

In the two tables below, I’ve summarized all of the above-mentioned experiments—their contexts and their consequences. In each table, the last few columns distill impacts on such outcomes such as investment, household spending (a key indicator of poverty), and “smoothing” (roughly speaking, the ability to handle financial emergencies). In these columns, “+” and “–” indicate changes up or down and “0″ means basically no change. Blanks mean no information is available. The last row of the savings table shows, ahem, the results that I would display if I were citing that “do not cite or circulate” study.

I hope that you, like me, are impressed with the diversity.
Credit

Savings

I note these patterns:

  • Except in Manila, wherever the impacts of microcredit on microenterprise (investment, profits, new business starts) were examined, they were positive. By and large, microcredit and microsavings do stimulate microenterprise. Of course the stimulus is statistical, not an iron rule. Not everyone who uses these financial services opens or expands a business.
     
  • There is hardly a sign that microcredit affects poverty. In Mongolia, those offered group microcredit (but not individual microcredit) spent more on food. In Bosnia and Herzegovina, those offered individual microcredit spent less. That’s about it.
     
  • But most of the studies look at impacts over 12–18 months. This leaves open the possibility that microcredit has bigger benefits only over the longer term.
     
  • Not evident in the tables is that all of the lending programs examined have been for-profit (I believe), except EKI in Bosnia and Herzegovina.
     
  • None of the microcredit studies has looked at the impact of combining credit with other services, such as classes in nutrition or accounting.
     
  • Like microcredit, microsavings stimulates investment in business activities, at least when in the form of a commitment account, one that makes it expensive or impossible to withdraw money before some specified date. This is particularly interesting because loans too can also be viewed a commitment device. Once you borrow the money, you are under a strong compulsion to set aside money for those payments, which is a kind of obligatory saving. Both commitment devices are stimulating investment.
     
  • But even within the short periods of these studies, savings is measurably boosting income and spending. This raises the question of why credit has not shown similarly rapid and positive impacts, and makes me a bit pessimistic that following up after a couple more years (as is being done in Hyderabad) will change the picture.
     
  • Finally, more with savings than with credit, there are hints that savings helps people sustain spending during important events such as serious illness or the birth of a child. This is just as you might expect. In Kenya, Dupas and Robinson found that in households offered savings accounts, spending fell less when someone in the family got malaria; but the effect wasn’t very significant statistically, which is probably why it has been dropped from the latest draft. The paper on Chile—preliminarily—finds something similar for a completely liquid account, one allowing deposits and withdrawals at any time.

A puzzle emerges. If, as Stuart Rutherford has long emphasized, savings accounts and loans are more similar and interchangeable than they might seem (being ways to assemble usefully large lump sums), and if commitment savings accounts are particularly like loans, why are the two producing markedly different impacts on poverty even within a year? An answer to that might lead to ways to make loans more useful to the poor. Perhaps the key, for example, is the greater flexibility of savings accounts. Even commitment accounts let you decide when to deposit. This would point to the value of flexible lines of credit, the most prominent example today being the Grameen Bank’s top-up loans.

To me, the emergence of such patterns, and the practical questions they generate for future research, demonstrate the value of this new, rigorous wave of research. I look forward to more.

Topics: 

Comments

Submitted by David Roodman on
Brett, I think that is a fair comment. What is being measured is not actually what happens to be people but what they report happening. Conceivably, the experience of using microfinance also exposes them to messages about how their lives *should* be going, which leads in turn to questions about how the experiment was implemented. Generally, I am sure the researchers are aware of this and do everything they can to prevent it. Note that the Kenya study actually did involve financial diaries. This is an expensive thing to do, which I assume why we haven’t seen more of it. But I hope we will. And at any rate, as you point out, everything you say applies to the older, non-randomized studies too. –David

Submitted by Jonny on
Hi David, Thanks for this summary. I am intrigued by the apparent divergence between the ‘investment’ and ‘wellbeing’ columns in the credit table above, as I would have thought that increased economic activity would have resulted in increased consumption spending. Do you have any ideas for what might explain that divergence? Several possibilities spring to my mind: 1. The businesses being established / grown are not earning the entrepreneurs any more profit. But in that case, are the ‘suppliers’ to these businesses enjoying greater revenues and income levels? 2. Increased business profits are being saved by the entrepreneurs, and not spent on consumption (seems unlikely). 3. Increased business revenues are being transferred to microlenders in the form of interest payments on the loans. Also, I know that many businesses in the developed world take several years to become profitable (look at Groupon for example!). Do you think the duration of these studies is long enough to enable us to draw firm conclusions? Imagine your answer will be “No, but it enables us to start to draw tentative conclusions”. Which I think is a good one! Thanks, Jonny

Submitted by Brett Hudson Ma... on
It’s great to see that the evidence emerging from RCTs is reaching a point where it can have real impact on practice. I sincerely hope that we will see more active integration of RCTs with financial diaries, as this is the best way to get the kind of granulated analysis that can really have impact. And I remain more than a bit concerned about implementation. David, you have described RCTs as the ‘gold standard’ for impact assessment. Without a doubt they are far superior to anything in the microcredit world that preceded them. But they still rely very heavily on some very old-fashioned tools: one-time questionnaires, in-person testimony, and in-person recall of complex transactional details. While it could be argued that these errors will balance out, results can also be significantly skewed by the propaganda that NGOs distribute in the field as part of their work. If I am told to expect that something will happen to me as a result of an NGO’s intervention, I may well concede that it happened (at least a little) when someone comes knocking on my door a year later to ask me questions. In socially-authoritarian societies people are often prone to saying what they think we want to hear. Is this being controlled for? One of the great strengths of financial diaries is that when people meet an interviewer over and, over, this effect washes out. I have to give credit to the researchers here by acknowledging that if impacts are being exaggerated, the effect is hard to discern. But it would be strange to see much in the way of sustainable impacts on a variable like household expenditure over a single year regardless of the intervention.

Add new comment