The short answer is, yes, they are.
I should clarify. I’m not saying randomized control trials don’t have their value. For specific tasks, they are pretty much the best at what they do. But too often, they are used — or suggested — by people who don’t entirely understand their limitations. And that’s where we start getting into problems with questions that we ask RCTs.
This post is the fourth in a series on randomized control trials. In case you missed the previous posts, you can start the series here: Have You Ever Seen a Unicorn? (Or a True RCT?)
Questions RCTs Can Answer
RCTs are excellent at answering questions about the average treatment effect of a simple experiment with the belief that change comes at the individual level.
For example: If I have a new cancer treatment drug, and I set up a randomized control trial where some patients receive my treatment and some don’t, the RCT can effectively tell me what the average impact of my treatment is:
- On average, does the treatment cure the disease?
- Does the average patient experience side effects?
- On average, how long does the treatment take to work?
Do you see the commonality between all these questions?
They’re all asking the RCT questions about the average effect on the average individual. But remember that thing I said at the beginning, that RCTs can answer questions about the average treatment effect of a very simple experiment with the belief that change comes at the individual level?
Unfortunately, this means that the fundamental belief that the causal impact is on the individual levels is deeply embedded within randomized control trials.
Questions RCTs Can’t Answer
If the causal impact (or other changed caused by your project) is happening at a community or societal level, your randomized control trial simply won’t capture it. (Or if it does, it will likely get the scale of effect size wrong.)
For example: If I have a new program that aims to reduce crime in my neighbourhood, and I provide my “treatment” to some members of a community and not to others, an RCT can’t necessarily measure the effectiveness of my program.
With programs that attempt to address broad issues like poverty or crime, we can’t place the onus on an individual for change. Asking RCT questions in this type of situation won’t provide answers that can hold up to what we understand about systems.
Some Things Aren’t Easily Quantified
When we ask RCTs questions, we’re looking at outcomes that are very simple and easily quantified (or at least, we should be). Often the results of RCTs are binary — a simple “yes, this worked,” or “no, it didn’t.”
Social practices, mental health issues, societal issues, and things like empowerment don’t fit this mold. These issues are more complex and can’t be condensed to fit into questions RCTs can answer. It’s critical to exercise care when applying data to social problems.
Are you struggling to understand what questions RCTs can answer for your organization? The team at Datassist is here to help. We work with social sector organizations of all sizes to help them leverage the power of data. We can help you too! Contact us today.