Are your survey questions measuring as precisely what you want to know as your bathroom scale does for you?
If your bathroom scale is not set up right, it can give you bad data, but if it is set up right, the response it gives will be simple, direct, and provide you what you need for decisions and next actions.
- The very first step in developing your questions is to determine the content, goals and purpose of each question, based on the goal of the overall survey.
As we discussed earlier in The Research Question, is the survey objective or goal to:
- describe characteristics, behaviors, events for a certain population?
- make a comparison between groups or outcomes?
- simply obtain feedback and learn about judgments or attitudes, interests, or opinions of a certain group or population?
The best method for judging whether the questions will result in the information you need to reach your goal is to, as Steven Covey is commonly quoted, “Begin with the end in mind.” Create the final data tables to evaluate whether your question results are on point, are extraneous, or are missing in gathering the information needed.
Another benefit of doing this evaluation in the beginning is that, when the survey goal and outcomes are clear to those conducting the survey, the surveyors can provide context in their survey preamble, which can effectively clarify and enhance the respondents’ ability to provide honest, accurate, and useful information.
In your analysis of whether your questions are achieving the goal, ask yourself:
- Is the question useful?
- Are several questions needed to ferret out the actual data needed?
- Do respondents have the information to accurately respond?
- Determine the best wording to avoid misinterpretation and get the information you are looking for.
Once you see if your survey questions achieve the goal, ask “will the question consistently produce the same outcomes” – can you rely on the information to be accurate? As described in the Survey Guide (a great tool from the University of Wisconsin Survey Center), survey questions should be simple, direct, and clear, with a single response option, and only a single subject and verb.
- Does the survey actually measure what you want to measure?
- Does the question mean the same thing to everyone, or does it need to be more specific? More general?
- Can the question be misunderstood?
- Are there assumptions that the question makes?
- Are there time constraints that might be unclear?
- Is the question inadvertently loaded to get a certain response?
- Avoid asking two questions in one. It is much better to split a long, complex question into two simple questions.
This one can be challenging as our normal conversations often include multiple part questions. “Does your mom and dad work weekends?” is a question we might easily ask a young person when exploring her home life. However, this question will not produce strong data. It needs to be divided into two questions, “Does you mom work weekends?” and “Does your dad work weekend?”
- Include a time frame in questions as often as possible.
For example, when conducting a survey on mental health, do not ask “How often do you feel anxious?” This is too broad of a question and you will get data that is not comparable across individuals. Instead ask “In the past week, how many times did you feel anxious?” The question from part three above would be improved by asking “How many weekends in the past four weekends did your mom work?”
Time frames need to be included in questions that are not obviously about time – such as places of residence, marital status, and income. It almost never hurts to be too specific.
- Choose the response format so that it matches the answer.
Structure the question and the form for the data you want to collect. If you want city and state, ask the respondent for these. If you provide response categories, make sure they are exhaustive, covering all the options.
Ensure that the scales and answer choices are very clearly defined. For example, if you ask a respondent to rate the quality of the service they received from 1 to 10, make sure you indicate whether 1 is for high quality service or low quality service. Do not assume that respondents necessarily think of scales in the same order as you do.
Research has found that scales up to seven are optimal. More than seven choices are too complex for most surveys and do not provide additional information.
Research has also found that when scale categories are labeled with words rather than just with numbers, respondents give more consistent and reliable answers.
- Put the questions in the optimal order.
Not only must the visual layout be simple, logical, and consistent, but the questions must be ordered to get the most accurate and complete information. For example, sometimes if it is a question where respondents might have difficulty recalling specific dates or situations, the surveyor can put the question later in the survey, after the respondent has begun processing and recalling. But if it is information they won’t have, consider who will actually need to be responding to the survey.
Also ensure that the questions are in an order that is as neutral as possible to the respondent.
Put any sensitive questions that may cause the respondent to end the survey at the very end of the survey. This often includes personal questions and demographic questions such as income or relationship status. If you must have this information to determine whether or not the respondent is eligible to participate in the survey, then include them at the beginning. In all other situations, include them at the end.
In this linked Survey Guide, you will find tips and checklists to work your way through an effective survey design. Datassist’s monthly Resource List news for subscribers www.idatassist.com contains more tools like this.
Datassist began this Survey Series earlier this month, with the #1 Survey Myth – Sample Size, followed up by a discussion of Excellence in Surveying. We’ve provided tools for calculating a sample size, and selecting a sample. You can come back each week to find new tools and tips in this series. We will take you from designing the survey tool, to actually selecting your sample, testing and timing, collecting and entering the data, and cleaning and auditing.[dismissable_alert class=”alert-info” ]Datassist is consistently providing real-world answers to unique data-based questions, and turns data into stories and pictures people talk about.
…So take time right now to: