I’ve spent a lot of time lately talking about RCTs (randomized controlled trials, for those of you who haven’t been listening). I’ve written blog posts. I spoke at the American Evaluation Association conference on RCTs. Even the latest issue of the Datassist newsletter was all about them. And for some of you, the messages I’ve been sending may be kind of confusing? Are RCTs really the gold standard for researchers? Is there a better alternative that your organization should be using?
I don’t blame you for the confusion. It’s a complex topic.
I like RCTs. They do what they do, really well. What I don’t like is the way some people insist on RCTs without really understanding why. RCTs are great for certain situations. But for many projects — including those you in the social sector are undertaking — they’re just not the answer.
Here’s my final (for now) on randomized controlled trials.
Sometimes RCTs are Great
The suitability of randomized controlled trials to your project depends on what type of question you’re trying to answer. While RCTs are very good at what they do, there are some serious limitations to what they can tell you.
Considering an RCT? Ask yourself these questions:
- Do you want to determine the average treatment effect for an entire group of people (not look at individual results)?
- Is the treatment or program you’re offering relatively simple and easily measured?
- Is existing expertise on your subject either non-existent or unreliable?
- Are the population and context of your trial similar to those you’ll use if the treatment or program proves effective?
If you can answer “yes” to these questions, and RCT will probably provide the information you’re trying to gather. Go forth, and randomly control your trials!
Sometimes RCTs are All Wrong
Basically, randomized controlled trials are really good at providing a snapshot of whether or not a program or treatment is effective. They allow some high-level evaluation of your efforts. But in many cases, you may want to know more than just “Yes, this seems to work,” or “No, it doesn’t.”
Unfortunately, RCTs are not so good at answering a lot of other impact evaluation or causal analysis questions. You may benefit from another data collection method if you’re looking for answers to any of these:
- Which treatment is most effective for individuals with this specific problem?
- What circumstances affect how this treatment or benefit works?
- Who is this treatment best suited for?
- Why did this program work (or not work) for these individuals?
There are also other ethical and operational issues that can crop up when attempting to run randomized controlled trials on human participants. (I’ve blogged about that before, so I won’t go into detail, but just remember that humans don’t always make great guinea pigs.)
Use the Right Tool for the Job
There’s a lot to be said for RCTs. They are a powerful tool — but it’s important to have the right tool for the task at hand. Hopefully, this post will help you determine whether or not an RCT is right for you. And if you’ve decided against an RCT, we’ve also talked about a few of the alternatives in the past:
- Regression Discontinuity Design
- Difference in Differences
- Propensity Score Matching
- Play the Winner
- Partially Randomized Preference Trials
Need help with an impact evaluation? Want to learn more about RCTs and what they can do? Get in touch with the experts at Datassist today. The most meaningful change begins with data.