A few weeks back, I introduced a new blog series on randomized controlled trials (RCTs) and why they might not be the best option for those working in nonprofits or the social sector. After a few minor delays (sorry!) I’m happy to bring you the next installment, where we’ll begin to examine RCT alternatives.
As we discussed in the first post, researchers generally run RCTs to be sure changes they are seeing are actually being caused by their intervention, rather than any number of potential outside influences that aren’t (or can’t be) measured. Obviously, in the social sector, we want to ensure our efforts are making a difference, but operational and ethical issues can arise when you are collecting data on real people, rather than string beans in a lab.
The next few posts in this series will cover RCT alternatives — many of which are often much cheaper and more accurate in measuring the impact of your team’s actions. These options are called quasi-experimental methods — they are similar to experiments (RCTs are experiments), but not quite the same.
The first RCT alternative we’ll discuss is matching and reweighting.
Weighting, of course, is something we’ve discussed previously and is an important part of data collection and analysis. Reweighting allows us to expand our analysis when we don’t necessarily have all the information we’d like about our subjects.
The matching part of matching and reweighting can be achieved in a couple of ways, but for today, I’d like to focus on one: PSM.
PSM (Propensity Score Matching)
Propensity score matching is essentially estimating the likelihood of an individual to be affected by your intervention and matching them to individuals with similar propensities.
OK, if you don’t have statistical inclinations, that probably didn’t clear it up much.
Basically, PSM lets you take a pool of data — on both participants and nonparticipants — and match them along the lines of key factors so you can test for different results between the people who most closely match, and better estimate the likelihood of impact on your participants.
Doing this allows you to analyze your data in a couple of different ways:
- You can use the matched cases to conduct a statistical analysis and examine the differences between treated and untreated individuals
- You can use propensity scores to weight your analysis to obtain a balanced sample of treated and untreated individuals
Putting PSM in Action
Say you’d like to measure the effect of participation in your literacy program on student success. Because students are all different, the likelihood of being able to compare your program participants to others who didn’t enroll in the program but share the exact same background and circumstances as your students is pretty low. That’s where PSM comes in.
To examine the impact of your program on participants (we’ll say those who do enroll in your program are “treated”), we need to compare them to a “control” group — people who aren’t participating in your literacy program.
We’ll start by examining factors that increase the likelihood of students to participate in your program — their parents’ age, income, and education level — and the effect of those factors on student success. By doing this, we can more effectively control for these differences to measure the actual impact of the program on participants, even though we don’t have a scientific control group to compare to.
- If all your program participants are from historically disadvantaged or less likely to succeed groups (young parents, low household income, no history of higher education), you can more safely assume that program participants who succeed were impacted by your efforts.
- If on the other hand, the majority of your program participants come from demographics that already predict success — from older, wealthier, or more educated parents — their success might not be attributable to your program.
Need Help Measuring Your Impact?
At Datassist, we understand that not everyone lives and breathes data analysis and visualization like we do. (Okay, almost no one does.) That’s why we are proud to work with data journalists and nonprofits around the world to help them collect, analyze and communicate data in an engaging, meaningful way.
Want to see what we can do for you? Check out some of our work with partners like FiveThirtyEight or Orb — or get in touch with us directly to discuss RCT alternatives, stats collection or data analysis and visualization.