There’s been a lot going on here with different posts from our online data journalism course and my time at the NICAR 2017 conference, but we haven’t forgotten about our series on nonprofits and randomized controlled trial (RCT) alternatives.
As a quick refresher, we’ve already talked about some of the pitfalls nonprofits using RCTs can run into, as well as one alternative method: propensity score matching (PSM). In this post, we’ll continue the conversation about ways to measure impact without relying on an RCT by talking about difference in differences.
What is Difference in Differences?
It sounds like a typo, I know. But when you stop to think about it, difference in differences is exactly what it sounds like. It involves the combination of two different analyses to create a single, more nuanced, analysis.
By combining time series analysis (analysis of different data points taken over equally spaced periods of time) and cross-sectional analysis (data from specific groups at a single point in time), difference in differences allows us to examine the differences between groups, over time.
But what does that achieve?
By comparing trends over time for one group with trends over time for another group, we are able to identify differences between the two groups. So for example, if you compared trends over time for a group of people involved in your organization’s efforts to the trends of an equivalent group not included in your project, the difference can be considered the impact of your project.
Using Difference in Differences
To use the difference in differences method, you’ll need to collect panel data — data gathered from the same groups of people (or string beans or whatever you’re measuring — but I think for most of our readers, it will be people) over time.
Using that data, you can examine how the difference between people affected by your project and people not included in your efforts change — does the difference increase? Decrease? Stay the same? The difference between those differences (hence the method name) is the impact you’re having.
Difference in Differences in Action
Let’s imagine you’re working on a project to provide breakfast to elementary school students. (After all, breakfast is one of the most important meals of the day — right up there with lunch, dinner, second dinner, tea, and elevensies.)
You want to understand the impact your program is having, but obviously, the scientific RCT approach isn’t really feasible. You don’t want to offer a free, nutritious breakfast to some children in a school and deny others, which is what you’d have to do to conduct a randomized controlled trial.
Using difference in differences methodology:
- You collect data on a large random sampling of students at the school
- Track which ones access the breakfast program and which don’t
- Analyze the differences between the two groups at the beginning of your program, and over time
Let’s say your goal is to improve math scores by providing nutritious breakfasts to kids who might otherwise not get one. You can compare the difference between the marks of children in the program and not in the program at the start of your project, and as time goes on — a change in the differences between the two groups indicates the impact of your efforts.
If students who elect to partake in the free breakfast program started with lower average math marks than those who don’t participate, but the scores of participating students move closer to the scores of non-participating students over time, you can extrapolate that your program is achieving its goal.
Datassist is Here to Help
Whether you’re a nonprofit trying to measure the impact of your efforts or a journalist trying to determine if the story you see is actually happening, the team at Datassist is here to help. If you’re struggling to tell your data’s story, count on us to:
- Assist with data collection, cleaning, and analysis
- Mindfully express complex data to your audience — whoever they may be
- Create captivating infographics, visualizations, and stories to communicate your findings
- Help you tell your story, the way you want it told
You’re working hard to make a difference. So are we. Get in touch with our team today to discuss how we can help you.