Subscribe To Our Newsletter

Get tips and tools to tell your data story better.

No, thanks

 In Data Analysis Concepts Simplified, Data Resources for Nonprofits

A lot of my posts talk about the importance of being scientific with data in the name of accuracy. Data can be a powerful tool for measuring the impact of your work and demonstrating it to others, but it also has the capacity to be incredibly misleading when handled carelessly.

Anyone with a scientific background will tell you that randomized controlled trials (RCTs) are the gold standard for gathering data. But for nonprofits, RCTs can pose a number of dilemmas; which is why I’d like to introduce our next blog series, Nonprofits and RCTs.

In this post, we’ll discuss some of the reasons why an RCT might not be the appropriate way to measure the impact of  your organization’s efforts. In future posts, we’ll outline and examine some viable alternatives and discuss which works best when.

Reporting Impacts: Not Always Easy

Lots of us need to show the work we do has an impact; that we are, in some way, using the resources at our disposal to improve life for someone, whether it’s by increasing incomes, decreasing discrimination, or offering solutions or alternatives to vulnerable members of our population.

As discussed in a number of our earlier blog posts on Data Analysis for Nonprofits, a growing number of policymakers and nonprofit organizations are transitioning away from reporting outputs or processes in favour of reporting outcomes and impacts. And an RCT is sometimes a good way to do this — but it is not the only way by any means.  

There are a number of barriers to the successful implementation of RCTs, and a range of issues that mean, even when they are possible, randomized controlled trials aren’t always the best option for data collection on socially innovative projects.

Ethical Concerns

While a randomized controlled trial can work brilliantly in a laboratory setting to measure the effects of, say fertilizers on green beans, nonprofits using RCTs will quickly run into ethical hurdles when trying to apply the same principles. You cannot ethically ask people to do something harmful to them, and when social good is your ultimate goal, denying aid or assistance to a significant part of a population in need is counterintuitive at best, and unethical at worst.

Of course, there is an argument to be made for applying scientific methods for the long-term greater good, but in many cases, withholding aid to a specific group can interfere with a project’s funding, as well as the political will to move forward with it.

Unbiased Implementation

In order to implement an RCT, your nonprofit’s staff will, in all likelihood, need to know who is assigned to each group — and that knowledge immediately invalidates your trial.

The point of a randomized controlled trial is to provide your organization with unbiased information on the effects of your efforts. If the people administering your programs are also monitoring your trial, how can the data gathered possibly be unbiased?

Splattering means the effects of your efforts don’t only show up where you intended.

Splatter Effect

“Splatter effect” (also referred to as “spillover”) is a broad term for the unintended consequences of your project — effects that occur in population members not directly involved in what you’re doing.

When you confine the benefit you are offering to a segment of the population in which you’re working, other members of the community are likely to be affected by the fact that they are not receiving the same benefit. Because they’re affected (albeit in a way you didn’t intend), these people are no longer valid participants in your control group. Unfortunately, splatter effect is basically unavoidable when attempting to conduct an RCT with humans.

Participants Seeking Alternatives

Maintaining control of your “control” group is critical to the validity of your nonprofit’s RCT. When you separate your population into two groups and offer your benefit (whatever it may be) to only one of those groups, it’s vital that members of the control group don’t receive some alternative to the benefit they are being denied. In a lab setting, this is easy enough; but it’s nearly impossible to stop members of the community from seeking other solutions to their problems — once again rendering them invalid to your control group.

This issue can rear its head in all types of nonprofit RCTs. If you’re offering a drug or medical treatment, control group members will seek alternative treatments. If you offer microfinance, they will look elsewhere for credit. If you’re offering job training, they will seek other avenues of advancement.

What’s the Alternative for Nonprofits Using RCTs?

Other issues with nonprofits using RCTs have been raised by others in our field, and we will explore them in more depth as we continue this series in the coming weeks.

At Datassist, our goal is to provide support and assistance to data journalists and nonprofits trying to convey the importance of what they do to peers, patrons and the general public. If you’re struggling to tell your story with the data you have, our team of statisticians and data visualization experts is here to help. Get in touch today.


Recommended Posts

Start typing and press Enter to search

Ignoring impact measurement for your nonprofit’s projects is like hitting the gym and never stepping on a scale.The ecological fallacy occurs when you try to tell data stories in the wrong direction.