Keeping a survey short is probably one of the most challenging tasks about survey design and administration. In my consulting practice, I am frequently approached with surveys that are dozens of pages long. When pilot tested, these surveys often take individuals several hours to complete. There are extreme cases where this length of survey is necessary and justifiable. But they are very rare. It is almost always better to collect less data well than to try to collect a large amount of data at one time from a single person.
In my research, after 20 minutes of answering survey questions, the quality of the data decreases by 11% in reliability. After 40 minutes of answering survey questions, the quality of the data decreases by a full 42%. This means that the data at the end of a long survey is almost worse than no data at all, because of its potential to produce misleading results.
The best way of reducing the size of your survey is to remove questions. Return to the core research question and purpose of your survey. It is quite common to experience scope creep when writing a survey and to unintentionally include a lot of questions that are close to your core research questions, but not actually about your core research question. This is particularly true when a group of stakeholders are designing the survey.
Go through your survey draft with three colored pencils – green, yellow and orange. Highlight in green each question that gives you data that is absolutely essential to answer your core research question. Highlight in yellow questions that you think will allow you to gain valuable insight into your core research question. Highlight in orange the questions that are good to have but you might be able to do without. Get your colleagues to do the same. Include only those questions that are highlighted in green by everyone. If your survey draft is still short enough, gradually add back the yellow questions.
Do a bit of research into what others have found using surveys in your field. Often there are one or two questions that are extremely strong predictors of what you’re trying to measure. Include these questions and drop the rest.
For example, a Harvard Business Review piece (The One Number You Need to Know, Frederick F. Reichheld) showed that the single question, “How likely are you to recommend X to a friend or colleague?” was as strong of a predictor of customer loyalty as any longer survey. This super short survey also provided actionable results for stimulating growth better than a longer survey that often ended up with low response rates and data that was misleading, ambiguous, and difficult to act on. This type of question works in the non-profit world too.
One way of reducing the length of a survey, when you simply cannot find a way to reduce the overall number of questions, is to divide the survey into several modules – one core module and then several adjunct modules. All survey respondents get asked the questions in the core module. And the survey field team rotates the adjunct modules through different respondents. This way you still get some data on all your questions. And the data you get is more reliable than if you asked all your respondents 2 hours of questions. Datassist has had good success with this method.
…So take time right now to:
- Sign up for monthly data science and visualization resources.
- Check out this recently published book that features Datassist work,
- Download this Datassist Road Map to a Great Survey
- Take a look at some of the work we have done.
- Friend us on Facebook for a personal connection with what we do.
- Follow us on Twitter for the latest news.
- Get inspired with us on Pinterest.