Over the last few weeks, I’ve been talking about randomized control trials (RCTs) — the good, the bad, and the ugly. I’ve been trying to address questions about RCT alternatives, and why I don’t, in fact, think we should rely on RCTs as our default. There are a lot of great ways to conduct impact analysis studies, and we shouldn’t relegate them to being just “RCT alternatives.”
We Don’t Need “RCT Alternatives”
One of the criticisms I often hear when talking about statistical methods that can answer both interesting and important questions is that the methods I propose make a lot of questionable assumptions. That I’m simply eliminating the assumptions made with an RCT with different — and in some cases, shakier — assumptions.
However, this criticism is rooted in the assumption that a randomized control trial could answer my key research questions. (Which they almost never can.) In the majority of projects I’ve worked on, I’ve needed impact analysis that includes answers to questions on:
- Specific impacts on people
- Different types of impact on different types of people
- What the project was best (and worst) for
- The longevity of the project’s impact
- Unexpected impacts that I didn’t plan for
What We Want from Impact Analysis
The point is that I’m not suggesting that any impact analysis methods I use are better than RCTs. They’re not. They’re different from RCTs. These methods can answer a multitude of questions that RCTs simply can’t answer.
As a rule, I usually want to get some combination of three main pieces of information from my impact analysis:
- Whether or not there was a causal impact
- What the mechanisms of that impact were
- An understanding of what changes took place and how they’re related
In short, I’m trying to identify more nuanced causality — something RCTs just can’t do.
Let’s Talk More About Causality
For my next blog series, I’m going to talk more about impact analysis methods that can help you identify causality, including some topics I’ve covered in the past, and some methods we haven’t mentioned here before:
- Propensity score matching
- Causation network models
- Serial replication
- Randomized roll-out designs
- “Play the Winner” randomization
- Interrupted time series designs
- Instrumental variables
We’ll also discuss different mechanisms of impact and how to understand the changes that our impact analysis uncovers. Stay tuned!
Of course, if you have specific questions about impact analysis or need expert assistance measuring or understanding the impact of your organization’s work, the team at Datassist is always here to help. Drop us a line anytime.