Given that we’re being told stories about polls and their results at every turn these days, let’s talk about the margin of error.
A story on the radio reports that a recent poll shows 75% of the town thinks the mayor is doing a good job. The announcer says the poll has a margin of error of 5%.
What does this mean?
You might guess it means the actual number of people in the town who support the mayor is between 70% and 80%. (75 + 5 = 80 and 75 – 5 = 70). Intuitively, this makes sense, as this is 75% plus or minus 5%. But usually this is not the case.
When working with numbers, statistics, and data, it’s important to take good care of yourself and your sanity by understanding the true concept that these numbers actually represent.
So What is the Margin of Error?
If you’ve ever hung up a phone on a pollster, you have direct experience with why the margin of error is just one source of the amount of difference between the survey results and the ground truth.
The margin of error usually measures how representative the survey sample is likely to be of the entire community. It assumes, among other things, the entire community was available to be included in the survey, but the whole population is never just waiting around to answer your questions. Most survey data does not have a strict margin of error that can be calculated exactly. Because of this, it is sometimes argued that online polls should not be said to have a true margin of error.
In reality, almost all media outlets report a margin of error and many readers expect to see the margin of error, regardless of how the data was collected. The general use of margin of error is to remind readers that all survey results have a level of uncertainty. They are never the absolute truth. And an estimated margin of error provides a general sense of the result’s overall accuracy.
More Like a Guideline
If you want to keep your wits about you when consuming survey results, it’s important to realize that the margin of error is usually giving you a general sense of how similar the people in the survey sample are to the people in the entire community.
The accuracy of survey results is calculated in complicated ways, and using the margin of error is one part of the equation. Other factors include calculating the errors caused by:
- People refusing to answer the survey
- Bias in survey question wording
- Mistakes made by the interviewers
If you want to get a general sense of how accurate the poll results you’re seeing actually are, read the fine print provided with the narrative. It will usually tell you about the (fake) margin of error, the sample size, and the way the respondents were contacted. If you really need to know how reliable the results are, contact the researchers directly to read the questions, ask about missing data, removal of outliers, and all the statistical nitty gritty.
Want to Know More?
At Datassist, we specialize in data analysis and visualization, and our goal is to help you make the most of your data. If you’d like to learn more about how your organization can leverage your data, get in touch with us (or subscribe to our newsletter) today.