One randomized control trial is never enough.
Continuing our series on why RCTs are kind of like unicorns, today I’m going to talk about the importance of conducting multiple RCTs — if it turns out that an RCT is even the right research method for you.
The results of a single RCT can’t reliably tell you whether or not your program has had a causal impact. To achieve that, you need to conduct multiple RCTs. Why, you ask? Let’s get started.
The purpose of a randomized control trial is to reduce or eliminate bias. What an RCT can’t do is eliminate variability. In fact, as bias in your research decreases, variability will inevitably increase.
If you want a solid level of variability and low bias, you need a very large sample size. And even then, it’s impossible to know if you’ve actually hit the target. Take a look at this illustration, which is commonly used in many books on statistics.
Are We Hitting the Target?
In the above image, we see a target. Imagine this represents the true value of the effect you’re looking for — in other words, the real amount of change brought about by your project. We don’t know what this number is. (In fact, we never will.) We are simply using data and statistical methodology to estimate it.
So in reality, the image should look like this.
That’s much less reassuring, isn’t it?
But it’s much more accurate in its representation of the real world.
Sample Size and Multiple RCTs
Let’s imagine your randomized control trial has a sample size and power that you believe will provide 90% accuracy in your results. That doesn’t mean you have a 90% chance of getting the true answer with a single RCT.
It means that if you ran multiple RCTs — let’s say 100 for this example — 90% of them would provide you with an answer that was in the true and correct range. The big downside here is that you have no way at all of knowing if the RCT you’ve just run is among the correct ones or if it’s one of the ones that completely misses the mark.
A one in ten chance of being entirely wrong is not nothing. Especially when your research has real-world implications — not just theoretical mathematical ones.
Back in the 1980s, the Minneapolis police department conducted a randomized control trial as they attempted to determine the best course of action when responding to domestic violence calls. After a single RCT, they determined that arresting the suspect was the most effective deterrent, and implemented a policy to that effect. Without further study, many other police departments across the country followed suit. The widespread adoption facilitated additional research of the strategy — and it was quickly found to be no more effective than other approaches.
Use RCTs Carefully
I’m not trying to argue that randomized control trials have no value. They’re an incredibly powerful tool when used carefully and for the right purpose. But it’s important to understand their limitations.
If you have questions about RCTs or need help measuring the impact of a program your organization is running, the team at Datassist is here to help. Get in touch with us today.