Information is not synonymous with knowledge. Information is only data, parts of the whole.
– Ruth Nanda Anshen
Do you know the difference between statistical significance and practical significance? Do you care? Are you thinking I’ve gone all data-nerd on you again? (I haven’t, I swear.)
Statistical significance simply means the statistic you’ve collected is reliable. It’s not random, and you can be confident it means something. In a nutshell, statistical significance is the absence of sampling errors.
Practical significance, on the other hand, is a much more subjective beast. It relies on factors other than statistical significance and is critical for decision-making.
Significance can be a valuable tool — if you understand what it is and how to apply it. Let’s dive right in, shall we?
Significance: Statistical vs. Practical
You’ve probably heard someone say something about how two groups differ in a way that is “statistically significant.” That isn’t just a figure of speech.
An elementary school wants to test different methods of teaching math to determine which is more effective. A common strategy is to try out one teaching method in one classroom, a different one in another class, and compare the results and performance of the two groups at the end of the term or school year.
If test scores in the two classrooms are different in a way that can be shown through analysis to not be random, we say the difference is statistically significant. Fantastic. We’ve shown one method to be superior and can implement it school-wide next term, right?
Not so fast.
What if the more successful teaching method is significantly more expensive than the alternative? Can statistical significance confirm the extra investment makes sense? Of course not. If you’re shaking your head and asserting that it can, try looking at it this way:
Let’s assume the method deemed preferable costs double what the other method costs, in extra educational materials and specialized training for the teacher. Schools don’t have unlimited budgets. The extra money has to come from somewhere — perhaps funding the superior math instruction program means eliminating school trips, or free nutritious breakfasts for underprivileged students, or new textbooks for other classes.
Statistical significance told us that one method was better than the other. Practical significance might show us that it is not so much better as to merit these kinds of sacrifices.
Effect Sizes and Significance
So if statistical significance doesn’t provide us with enough information to base a decision on, does that mean we should collect data, analyze it, and then throw it out and go with our guts anyway? Not quite.
There is a way to measure how much the average of the first math class differed from the second. It’s called effect size, and it can help us determine how much of a cost increase (if any) is reasonable.
Effect size lets us quantify the difference between the two groups — if the effect size is small, the two groups are very similar. Effect size emphasizes the size of the difference, rather than simply letting us focus on whether or not a difference exists.
Back in our math classes, the effect size lets us see how much of an effect the competing instructional methods had. If the doubly expensive method produced a 5% increase in the class average, we might determine that cutting resources from other programs is not a worthwhile investment. On the other hand, if one method costs twice what the other does but doubles the performance of our math students, it might be reasonable to re-examine our budget.
(It is important to remember that, even if the effect size is large, the two groups may still have a great deal of variation within them — meaning not all members of one group will be different from members of the other.)
Need Help Determining Significance?
If you or your organization are struggling with significance, effect size, or any other aspect of your statistical analysis, the team at Datassist is here to help. We work with nonprofits, journalists, and policymakers around the world to help them leverage data in ways that educate and engage. Get in touch now to see what we can do for you.