To conclude our blog series on alternatives to randomized controlled trials for nonprofits, today’s post will introduce another useful method nonprofits can use to measure impact: regression discontinuity design, or RDD.
In case you missed any of the previous posts, we’ve already covered why nonprofits need an alternative to RCTs and a couple of options: propensity score matching (or PSM), and using difference in differences methodology.
We’ll start at the beginning (because it’s a very good place to start):
What is Regression Discontinuity Design?
Regression discontinuity design — sounds complicated, right? The name is enough to scare off anyone considering RCT alternatives. But in reality, RDD is a simple and straightforward tool that can prove very useful in measuring the impact of your organization’s efforts without conducting a formal experiment. To be honest, its name is by far the most complex thing about it.
So how does it work?
The idea behind regression discontinuity design is to generate a consistent estimate of your impact using a discontinuity in the level of treatment related to something observable. More succinctly, you’ll compare those just above the threshold of eligibility in your program (whatever it may be) with those just below it.
RDD is the ideal choice when your project or initiative is being offered to a group of people who meet a certain qualification that is measurable on a continuous scale — essentially any metric that is a number. Using that characteristic, you can synthesize a control group for your project, without actually denying access to qualified applicants.
Creating a Control Group
The easiest way to illustrate how regression discontinuity design works (without sounding discouragingly technical) is by example:
Let’s say you’re running a program that offers nutritional support to low-income households. In order to qualify for your program, households must have a demonstrably low income — for this example, we’ll say you’re offering assistance to households with an annual income of $20,000 or less.
As we’ve discussed in previous posts, the most scientific way to measure your program’s impact is to measure the change in households you help against households that don’t receive assistance.
But of course, you don’t want to deny anyone in need access to your program — it would be counter to your goals, and in many cases, unethical. So how do you get a control group to which you can compare your participants?
Using regression discontinuity design, you will create a pretend control group by collecting data on households that have incomes just over $20,000. These families don’t quite make the cutoff to receive the benefits you’re offering, but in reality, are facing the same challenges as the group you are working with.
Measuring Your Impact
So now you have your participants and your control group. Since you’re running a nutrition program, let’s presume your standard of measurement is each household’s Food Security Index score — with the assumption being that your efforts will help improve the score of those in your program. You’re ready to start measuring impact.
- Collect FSI data from participating households (incomes less than $20,000) at the beginning of your project
- At the same time, collect FSI data from non-participating households with incomes close to the program threshold (say, between $20,000 and $24,000)
- At the end of your program (or at whatever milestone you choose to begin measuring impact), collect FSI data on both groups again and compare
Regression discontinuity design works in this case because the cutoff point of $20,000 is not meaningful in a predictive way. A household with an income of $20,000 will be very similar to a household with an income of $20,100, and a lot like a household with an income of $20,800, even though the latter two don’t meet the qualification to participate in the program. RDD lets you take advantage of this fact — you can use the disqualified households as a constructed control group.
Of course, this methodology doesn’t have to be restricted to projects where eligibility is determined by income — any time participation or benefits eligibility is determined using a number (continuous eligibility index), regression discontinuity design can be applied. Test scores, poverty scores, age, and income are just a few of the most obvious examples.
Need Help Communicating Impact?
At Datassist, we do more than just create compelling data visualizations. (Although we do love data viz!) If you’re a nonprofit or journalist struggling to access, collect, analyze or communicate data to tell your story, our team is here to help. We specialize in data science for nonprofits and journalists and can help you convert data into stories and pictures that will move your audience. Get in touch with us today.