The idea of weighting survey results is a sticky one — that many people dislike and view as inconvenient. After all, you designed a great survey. You spent big bucks on fancy sampling. Who wouldn’t want to be able to simply take that data, estimate results, and make a decision?
But weighting survey results is, unfortunately, a necessary step in the process. And you’ll need to do it just about any time you want to use your survey sample data to make statements about a larger population.
What Do Survey Weights Do?
At the most fundamental level, we use weights to adjust our sample. We stretch it in different ways so that our results more accurately reflect the population we’re studying.
Why do we call it weighting survey results then?
The term weighting refers to the different amounts of load or emphasis we give to each individual survey response. Responses from a type of person who is very rare will get greater emphasis in the final results. In contrast, respondents who represent common types of people will be de-emphasized.
Different Weighting Methods
There are a number of different ways of weighting survey results. We’ll review a few of them here quickly, just to get you started. (We’re not going to go into the actual calculation methods for each one — just explain what they mean.)
Choosing what kind of survey weights to use isn’t an either-or prospect. Different weighting methods can be used — and are often needed — on the same survey results. For example, if we conduct an opinion poll to help us understand our district’s attitude toward a new policy, we might need several different weights to ensure our results accurately reflect the population’s sentiments.
We use design weights to account for the different probabilities of being sampled that different respondent types have. Using our example above, let’s say we’re collecting data based on a list of addresses. People who live in a place where many families share the same address will have a lower chance of being surveyed than people who live at single-family addresses. Weighting our survey results ensures our results won’t be skewed by this discrepancy.
You can use non-response weights to correct for the fact that some types of people are less likely to be willing to participate in your survey than others. To illustrate, let’s imagine that young people in our district are less inclined to answer our survey questions. Weighting our results ensures that we account for this fact by placing more load on the responses from young people who do participate.
Most commonly, when people talk about survey weights, they’re referring to calibration weights. This does not, however, mean that calibration weights are the most important way to weight survey results. (Full disclosure: calibration weights are the easiest to calculate, so they’re the ones people pay the most attention to.)
You can use calibration weights to make the characteristics of your sample closely match the characteristics of your population. This is commonly done using demographic data (like gender, age, income level) that is publicly available (from a Census, for example) as the target and adjusting the sample demographics to match that target. This method of weighting survey results is sometimes called post-stratification weighting.
Need Help Weighting Survey Results?
Datassist works with journalists, nonprofits, and social sector organizations of all sizes to help leverage the power of data to tell your story. If you need help collecting, weighting, analyzing, or visualizing your data, we can help. Get in touch with our team today.