How to write survey questions

Statistics Canada has a worked example that should be pinned above every survey draft. Asked whether they approved of “Sunday shopping”, 73 per cent of respondents said yes. Asked whether they approved of abolishing “the Sunday pause day”, 50 per cent said yes. Same policy, same population, 23 points of difference produced entirely by the words.

That is the whole problem. A survey does not measure opinion, it measures opinion as filtered through your wording, your options and your order. The craft is in reducing the gap between the two.

Start from the decision the survey feeds

Before drafting a question, write one sentence: “when the results come back, we will decide X”. If you cannot finish that sentence, you are collecting data rather than running a survey, and the questions will sprawl to match.

One decision per survey, two at the outside. The moment a second team adds “while you are asking, could you find out about…” you have a longer survey with a lower completion rate and answers from a self selecting minority patient enough to reach the end. Two short surveys sent a month apart beat one long one.

The decision also fixes the population. A question about the checkout experience only means something asked of people who reached the checkout, which usually means the survey is triggered rather than broadcast.

The three wording faults

Leading. The question tells the respondent which answer you want. “How would you rate our excellent new booking system?” has done the rating already. Strip the adjective and let the scale carry the judgement. Statistics Canada’s Sunday shopping example is the same fault dressed as a policy question.

Double barrelled. One question, two things. “Was the room clean and comfortable?” cannot be answered by someone who found it spotless and cramped. Statistics Canada’s version is “do you plan to leave your car at home and take the bus?”. If a question contains “and” or “or” between two nouns, split it.

Loaded. The question smuggles in an assumption. “How long have you been unhappy with the reporting?” assumes unhappiness. So does “what is the main reason for your score?” if the score was a 10, which is why that question is best worded neutrally rather than as a complaint prompt.

One more from Pew that catches experienced writers: agree or disagree formats invite acquiescence bias, where respondents tick agree because agreeing is easier than disagreeing, particularly among people with less formal education. A forced choice between two stated positions gets a truer answer than “do you agree that…”.

Eight question types and the field that renders each

Survey guides list question types. Form builders list field types. They are not the same list, and the mapping is where most drafts go wrong. The table at the end of this article sets out the eight types you actually need and the field each one becomes here.

Three things in it are worth saying plainly.

There is no star rating, slider or opinion scale field. A rating scale is radio buttons with labelled options, or a single row of a rating matrix. A 0 to 10 recommendation question is 11 radio options, which is exactly how the example survey builds it. That is not elegant, but it is honest, and it exports cleanly.

There is no ranking field. If you need one, the nearest workable thing is a matrix with rank columns, and nothing stops a respondent using the same rank twice.

A number field in a conversational form does not enforce its own range in the browser. The minimum and maximum are checked when the form is submitted, at the end, after every question has been answered. For a survey, avoid tight numeric ranges in the conversational format and use a choice question instead.

Across the 55 example forms on this site, 890 input fields break down as 245 short text, 162 single choice, 111 long text, 74 multiple choice and 72 dropdown, with only 10 rating matrices in the whole set. Choice questions carry most surveys. Matrices are the exception, not the backbone.

Scales

Use five points for satisfaction and agreement. Label every point rather than just the ends, so that “3” is not left to mean whatever the respondent decides it means.

Keep a neutral midpoint when neutral is a real position, and drop it when it is not. An odd numbered scale gives people somewhere to hide; an even numbered one forces a lean. Neither is wrong, but decide deliberately.

Never mix directions. If one question runs from very satisfied to very dissatisfied, every question runs that way. Reversing a scale halfway down a matrix to catch inattentive respondents mostly catches attentive ones on a phone.

And once a scale is in use, leave it alone. AAPOR puts it best: if you want to measure change, do not change the measure.

Order and length

Start easy and interesting, which both Pew and Statistics Canada recommend, and which mostly means starting with the question the respondent already has an opinion about. Group by topic. Put demographics last unless they screen for eligibility, because they are the least engaging thing you can open with.

Sensitive questions go late, when the respondent has already invested effort. Question order also changes answers through contrast and assimilation effects, so if you rerun the survey, keep the order identical.

Length is best handled by format. A conversational form shows one question per screen, which makes six questions feel like six small asks rather than one wall. The 16 conversational examples on this site have a median of about 10 fields; the 39 classic ones have 18. The format quietly disciplines the length.

Open questions

One open question, near the end, optional.

Open questions produce answers you could never have listed. Pew’s illustration is a 2008 poll where 58 per cent picked the economy from a list of options and only 35 per cent volunteered it unprompted. The list changes what people say.

Give the open box a prompt rather than a blank. “What is the one thing we could do better?” collects something usable. “Any other comments?” collects white space. And keep it to one, because open text exports as a single column and someone has to read every row.

The finished survey

The example below is six questions: a five point satisfaction scale, a 0 to 10 recommendation question built as 11 radio options, a required open question asking the reason for that score, an optional multiple choice on what went well, an optional open question on what to improve, and an optional email for a reply.

Three questions are required and three are not. The open question about improvement sits last. Nothing asks two things at once. It reads in about ninety seconds, which is roughly the ceiling for a survey nobody was expecting.

For a matrix of statements rather than single questions, the rules change slightly, and the rating matrix note covers the one answer per row decision. For forms where the questions are set by law rather than by you, see patient intake form requirements.