We’re big fans of RCTs — randomized controlled trials — here at Datassist. Really, most statisticians are. RCTs are considered the gold standard of research. That said, we’re also acutely aware of some of the issues that come with relying on RCTs — which is why the idea of “Play the Winner” is so interesting.
What is it?
“Play the Winner” is a cool alternative to RCTs. It combines an element of randomization with some informed human knowledge. It’s particularly useful for addressing some of the ethical issues that can crop up when using RCTs to look at humans.
If you’re testing which kind of fertilizer makes green beans grow faster, there are no real ethical concerns. There is nothing unethical about denying some plants intervention in order to compare results. But what if you’re testing a new drug on cancer patients? Or trying to determine if a food aid program you’re running improves health in a low-income group? Withholding something that could drastically improve people’s lives puts you in a sort of morally untenable position. That’s where “Play the Winner” comes in.
You… Play the Winner
“Play the Winner” is an adaptation of standard randomization techniques. In this method, the options are still randomized, but, as one option proves to be superior to others, the odds of receiving that option increase. For example:
Imagine you’re providing food aid to low-income families, with the goal of improving children’s health. In Option A, families receive a weekly food allowance and in Option B, they receive a single lump-sum payment to help them purchase food. You draw letters (A or B) from a hat to determine which option each family will receive.
For the first family, there are two letters in the hat: A and B. You draw one and apply it to that family. If that family shows an improvement, a second letter A is added to the hat with the original, meaning the odds of the next family receiving an A increase. If no improvement is shown, a letter B is added to the hat with the original A, meaning the odds of the next family receiving an A decrease.
As the process continues, the odds of families receiving the more effective option continue to increase. This means that, although the trial is still random, fewer people are deprived of a program that could effect positive change in their lives. The “Play the Winner” strategy reduces the ethical challenges of standard RCTs.
Is There a Downside?
This sounds great! Why don’t we always use “Play the Winner” in place of standard RCTs?
There are some disadvantages. Sample size calculations are challenging at best using this method. In addition, the process can be time-consuming. In the example above, the outcome of each family must be determined before the next family can be added. Not only does that delay the assistance those families need, but it also makes the study a very lengthy process.
We can modify “Play the Winner” randomization to make it more practical. Instead of waiting for the outcome of each individual entry to determine the odds for the next one, we can look at results in stages: we could start the program with 50 families, and use their outcomes to determine the odds for the next 50, and so on.
And of course, this method is still a form of RCT — which means it has the same weaknesses. Measuring averages can be a great way to determine the effectiveness of a program. But it won’t provide you with real insight into individual results until you drill down a bit. Participants may also seek alternative solutions to their problems during your trial — or while waiting for their turn.
Want to Know More?
At Datassist, we specialize in helping nonprofits, journalists, and social sector organizations leverage the power of data. Do you need help measuring impact or interpreting your data? Our experts are here to help. Get in touch with us today and let us help tell your data story.