I’ve spent years working with social sector organizations, helping them answer important questions with data. And the idea of statistical power is one that’s often thought of too late — or not at all.
Why does it matter? If you hope to understand how likely it is that your research noticed an effect or trend actually occurring in the real world, you need to understand the concept of statistical power. Evidence-based decision- and policy-making is gaining popularity. If you can wrap your head around the idea of statistical power, you’ll better understand your chances of making the right choice based on your data.
What is Statistical Power?
There are a few different ways to think about statistical power. All of these are essentially correct. Statistical power is the probability of:
- Rejecting a null hypothesis that is actually false
- Accurately choosing to reject a null hypothesis that is false
- Failing to reject a null hypothesis that should be rejected
- That a test of significance will detect an effect that is actually present
- That a test of significance will detect a deviation from null, if one exists
OK, that was a lot of statistical jargon I didn’t really get, Heather. What the heck does this mean?
What Statistical Power Looks Like
To better explain statistical power, let’s look at an example.
Say we’re doing some research to determine whether or not our government’s program of providing food supports to selected households has improved the school attendance of children in those households. There are a few possible combinations of what is actually happening in the real world and what our research shows:
- The extra food did make a difference in school attendance and our research shows that the extra food did make a difference in school attendance. (This is a correct finding of the research.)
- The extra food did make a difference in school attendance and our research shows that the extra food did not make a difference in school attendance. (This is an incorrect finding – Type 2 Error.)
- The extra food did not make a difference in school attendance and our research shows that the extra food did not make a difference in school attendance. (This is a correct finding of the research.)
- The extra food did not make a difference in school attendance and our research shows that the extra food did make a difference in school attendance. (This is an incorrect finding – Type 1 Error.)
Getting the Right Combination and Avoiding the Wrong One
Statistical power refers to the probability of getting the first combination right and avoiding the second. It’s how likely it is that your research will actually detect an effect or trend when that effect or trend is occurring on the ground.
But what about combinations three and four?
This is the tricky bit. It’s important to note that statistical power doesn’t tell us anything about the third or fourth combination. Statistical power doesn’t tell you that will accidentally avoid thinking your project made a difference when it didn’t. And, most crucially, it can’t give you confidence that there is no trend if no trend exists. (Complicated, I know.)
Determining Your Work’s Statistical Power
There are four basic elements of your research that will help determine how high your work’s statistical power is.
- Significance level (or alpha)
This is the p-value or confidence level you’re using in your work. A level of p<0.5 is common practice, but there is no real meaning to this number.
- Sample size
This is the number of people (or units) you’re collecting data from.
- Variability (or variance) in the measured response variable
This is how much of a difference there is between each individual (or unit) you’re trying to understand.
- Magnitude of the variable’s effect
This is how large or small the actual difference made by your project is.
Understanding what statistical power is and how it applies to your research is a critical part of using data — especially if you’re using that data to make life-changing decisions or policies. Need help ensuring your data says what you think — and what you think is really happening?
Talk to the experts at Datassist now.