Subscribe To Our Newsletter

Get tips and tools to tell your data story better.

No, thanks

 In Data Analysis Concepts Simplified, Experts

This is the second installment in my blog series about RCTs and the problems associated with them. It started last week with Have You Ever Seen a Unicorn? (Or a True RCT?) and was inspired by conversations I had recently at the Good Tech Conference in Chicago.

Prompted by an audience question, I spent a bit of time during my presentation at the Good Tech Conference talking about why true randomized control trials are rare and why the misunderstanding of what constitutes an RCT causes so many problems. Farther along into the workshop, I got another question:

“Yes, but what are some RCT alternatives for proving causal impact?”

A light bulb went off in my head.

 

RCTs Aren’t the Best at Measuring Causal Impact

I’ve been accepting the premise of that question for far too long. (We all have, really.) A big part of the narrative that has so many people convinced that RCTs are the “gold standard” of impact evaluation is the belief that they demonstrate causal impact. And guess what? They don’t really measure lots of kinds of causal impact.

  • Causal impact indicates that something is happening (or has happened) because of something else that is occurring (or occurred). At its most simplistic: one thing caused another to happen/succeed/fail/etc.
  • Randomized control trials demonstrate the unbiased average effect a treatment (be it medicine, financial aid, or social program) on a population. It doesn’t inherently prove causality; it measures how an event (or treatment) affected a population on average.

There are many other kinds of causal impact that cannot be measured by RCTs. When faced with issues relating to randomized control trials, we must seek out RCT alternatives to measure causal impact in much more meaningful ways.

 

Different Methods, Different Purposes

Let’s stop accepting the flawed premise of these question. We should stop justifying our use of “RCT alternatives” — because it makes about as much sense as asking someone to justify why they’ve chosen pistachio ice cream over strawberry. They are two very different things, and they serve two different purposes.

There is no real alternative method if you need to get an unbiased estimate of the average effect of a treatment on a population. That is always a job for an RCT. Always.

But if your causal impact question is a different one, you need a different method. It’s not an RCT alternative, because an RCT was never a viable option. What you need is simply a different — and no less valuable — research method.

Does it really matter how your data is collected? (Yes!)

Questions an RCT can answer:

  • What was the average effect of our project on household incomes in this community?

Questions an RCT cannot answer:

  • What was the causal impact of our project on the typical household income in this community?
  • What was the trajectory of the causal impact of our project on women over ten years of activity?

If your question is about anything other than the average treatment effect on a population, you can’t use a randomized control trial. You need another method. Period. So let’s stop looking for “RCT alternatives” that can answer our causal impact questions. These methods aren’t related to RCTs in any way. And the sooner we stop accepting the premise that they are, the sooner we’ll stop pushing RCTs as the “gold standard” of impact evaluation.

Next week, I’ll continue this series on RCTs and talk about how to use them correctly. Stay tuned.

Do you have questions about RCT use? Do you need help measuring causal impact for your organization? The team at Datassist is here to help. We work with journalists, nonprofits, and social sector organizations of all shapes and sizes. Get in touch with us today.

This post is the second in a series on randomized control trials. Keep reading here: Are Multiple RCTs Better Than Just One?

Recommended Posts

Leave a Comment

Start typing and press Enter to search

If treatment providers know who is in the control group and who is in the treatment group, you aren’t conducting a true RCT.Graphs with targets removed show scattered dots with no discernable pattern.