Subscribe To Our Newsletter

Get tips and tools to tell your data story better.

No, thanks

 In Data Analysis Concepts Simplified, Data Resources for Nonprofits, How To

I’ve spent a lot of time lately talking about RCTs (randomized controlled trials, for those of you who haven’t been listening). I’ve written blog posts. I spoke at the American Evaluation Association conference on RCTs. Even the latest issue of the Datassist newsletter was all about them. And for some of you, the messages I’ve been sending may be kind of confusing? Are RCTs really the gold standard for researchers? Is there a better alternative that your organization should be using?

I don’t blame you for the confusion. It’s a complex topic.

I like RCTs. They do what they do, really well. What I don’t like is the way some people insist on RCTs without really understanding why. RCTs are great for certain situations. But for many projects — including those you in the social sector are undertaking  — they’re just not the answer.

Here’s my final (for now) on randomized controlled trials.


Sometimes RCTs are Great

The suitability of randomized controlled trials to your project depends on what type of question you’re trying to answer. While RCTs are very good at what they do, there are some serious limitations to what they can tell you.

Considering an RCT? Ask yourself these questions:

  • Do you want to determine the average treatment effect for an entire group of people (not look at individual results)?
  • Is the treatment or program you’re offering relatively simple and easily measured?
  • Is existing expertise on your subject either non-existent or unreliable?
  • Are the population and context of your trial similar to those you’ll use if the treatment or program proves effective?

If you can answer “yes” to these questions, and RCT will probably provide the information you’re trying to gather. Go forth, and randomly control your trials!

RCTs work well for some times of evaluation, including impact and program evaluation.

Sometimes RCTs are All Wrong

Basically, randomized controlled trials are really good at providing a snapshot of whether or not a program or treatment is effective. They allow some high-level evaluation of your efforts. But in many cases, you may want to know more than just “Yes, this seems to work,” or “No, it doesn’t.”

Unfortunately, RCTs are not so good at answering a lot of other impact evaluation or causal analysis questions. You may benefit from another data collection method if you’re looking for answers to any of these:

  • Which treatment is most effective for individuals with this specific problem?
  • What circumstances affect how this treatment or benefit works?
  • Who is this treatment best suited for?
  • Why did this program work (or not work) for these individuals?

For some kinds of evaluation, there are many better alternatives to RCTs.

There are also other ethical and operational issues that can crop up when attempting to run randomized controlled trials on human participants. (I’ve blogged about that before, so I won’t go into detail, but just remember that humans don’t always make great guinea pigs.)


Use the Right Tool for the Job

There’s a lot to be said for RCTs. They are a powerful tool — but it’s important to have the right tool for the task at hand. Hopefully, this post will help you determine whether or not an RCT is right for you. And if you’ve decided against an RCT, we’ve also talked about a few of the alternatives in the past:

Need help with an impact evaluation? Want to learn more about RCTs and what they can do? Get in touch with the experts at Datassist today. The most meaningful change begins with data.

Recommended Posts

Start typing and press Enter to search

Inclusivity in design is about more than just not being overtly racist.