Using Randomized Trials for Product Design

People are unpredictable. We don’t always act in rational ways, and there is often a difference between what we say we would do in a particular situation and what we actually do when confronted with those circumstances.

This unpredictable nature makes it tricky for businesses and non-profits looking to increase demand for their products and services. Even the best-run focus groups, which provide a space to discuss a product with people who would normally be interested in it, can be misleading. How can a message conveyed by such a small group of clients truly reflect the larger population? The truth is, it probably can’t, especially when individuals can’t accurately predict their own behavior in a particular scenario. So what is a marketer or a business to do?

Randomized experiments are one potential answer. Although randomized experiments are most often thought to be the tools of academic researchers, these tools are being used in a variety of contexts by people who have no research background or training. The basic idea is to test different approaches on groups that are as similar as possible and to see how each approach effects people’s behavior. The key is to use the randomization to ensure that the test groups are close to identical. If the groups are identical, then when you try approach A on a group and approach B on an identical group, then any differences between the groups can be attributed to the difference between A & B. This approach is widely used in medicine, for instance, by giving some patients a particular treatment and giving others a placebo.

This concept can be applied to work in businesses as well. In a recent paper, researchers worked with a microfinance institution in Jordan to test different types of advertising messages on people interested in taking out a loan. The messages would sometimes change one sentence in the advertisement which would sometimes be enough to lead to changes in the proportion of individuals who were interested in applying for a microloan. Since changing the words that an organization says to potential clients or beneficiaries does not cost an organization money, optimizing the advertising message could be a very cost-effective way to improve the performance of different activities.

A similar strategy can be used to test different product designs against each other. For example, in the work in Jordan, researchers also compared the demand for a normal microloan to a sharia-compliant microloan. There was an open question regarding if demand for sharia-compliant loans was as strong as demand for conventional loans, and using a randomized control trial, the researchers found that demand for the sharia compliant loan was actually higher than demand for the conventional alternative.

This technique can be used widely, and what it requires is careful implementation of randomizing the things that the organization is interested in testing. For example, if an organization wants to check to see if putting a longer deadline on an opportunity (e.g. "Apply before the end of the month to save your spot!"), the organization needs to randomize which of their potential beneficiaries will get the opportunity with the deadline and which will not. This can be easy if done through things like e-mail, or normal mail or SMS. Then the organization will need to follow up to see what proportion of the group that go the deadline applied and compare it to the proportion of the group that applied of those who did not get the deadline. Similarly if an organization wants to compare demand for two different products, they can randomly allocate their clients to different groups and then offer on product to the first group and the other product to the second group.

These strategies can help an organization learn more about their own work, and improve upon their offerings, from both an advertising and product side, without costing the organization much. If the right tools are available, randomization can lead to a culture of continuous learning and improvement.

Read this post in Arabic on the FinDev Gateway