The Trials and Tribulations of Randomized Control
September 4, 2008, Evelyn Stark
Randomized Control Trials (RCT) have recently been touted by some in the microfinance industry as the best (or only) way to truly measure impact of microfinance. Considering the dearth of any other “proof” that microfinance helps poor people fight poverty, it’s clear why randomized control trials hold so much appeal: It’s scientific, uses a counterfactual/control group (which some of our prior studies didn’t do at all) and has the attraction of a definitive answer. It doesn’t hurt that the economists running these trials often manage to come up with some pretty interesting and fun studies, to wit: Dean Karlan, Jonathon Zinman and Xavier Gine's recent paper - “Put your money where your butt is: a commitment savings account for smoking cessation” .
But, our question is: are we looking to RCTs to do too much? And, by leaning towards over-reliance on RCT, are we forgetting about the need to use the right tools for the right issues? Are we in danger of throwing away “OK” monitoring, evaluation and impact methods and tools instead of improving them to be more useful and accurate?
The 12th Economist article entitled “Control Freaks” was clearly skeptical about randomized controls and led me to very unscientifically survey some colleagues, and Google. I found the following issues elicited the most concern with randomized control trials (note -there were no randomistas amongst respondents – they may have answers for each point!):
• Difficulty in setting up a randomized trial (client selection, staffing, follow-up);
• Expense - cited as equal to the cost of the microfinance intervention being tested
• Ethical challenge - the control group is “excluded” from service for the period of the test
• Time – results are measured over a relatively long periods (at least 1 year) and therefore the inability to make changes to the product/service in that time
• Relative merit (time, cost & results) to good market research and product piloting seemed unclear, or negative
• Limited by the specificity of the research question; unable to probe nuance and context or make results replicable across programs in different contexts
It seems that the last three points in particular make a strong case against putting all of our efforts into RCT in today’s dynamic, competitive microfinance environment. As A. Bradford Hill “one of this century’s foremost biostatisticians” said (in 1965): “All scientific work is liable to be upset or modified by advancing knowledge”[1].
As our industry innovates and expands, can MFIs afford to lose a competitive edge while narrow questions are tested; And, as an industry, can we afford to forego the rigorous, impact analysis that RCT provides?
[1]Quoted from: “Preventing HIV Transmission: the role of sterile needles and bleach.”
