How to pilot a survey before you send it

To pilot a survey, send the finished draft to about five people who match your audience, time how long each one takes, ask them to think aloud or explain each answer afterwards, and fix every question that someone misread or could not answer. Then run one or two more testers on the fixed version. It takes an afternoon and saves a survey that would otherwise collect the wrong answers.

Why pilot at all

A question that is clear to the person who wrote it is often unclear to everyone else. The Nielsen Norman Group puts it plainly in its guidance on pilot testing: tasks that seem perfectly clear to you and your team can be confusing or misleading to participants. A survey has no facilitator to clear up confusion, so every misreading turns into a wrong answer that looks like a real one in the results.

Large statistical agencies treat this as mandatory. The US Census Bureau’s Statistical Quality Standard A2 requires pretesting whenever a new questionnaire is developed, when questions are revised because data quality was poor, and when an existing instrument changes substantially. It names the accepted methods: cognitive interviews, focus groups where participants complete the questionnaire first, usability techniques focused on understanding, behaviour coding, respondent debriefings and split panel tests. It also requires that pretests use people from the survey’s own population.

Step 1: choose five testers

Pick people who match the audience: two customers, a supplier, a new member of staff, whoever the survey is for. Avoid the colleagues who helped write it except for a first check of typos and jumps. Five is a practical number for a small survey; the Census standard requires pretesting but sets no minimum, so the right number is the one that stops new problems appearing.

Step 2: time it

Note the start and finish of each tester. If the welcome screen promises three minutes and testers take six, change the promise or cut questions. The how long should a survey take post compares promised times with measured ones.

Step 3: think aloud, then debrief

Ask testers to say what they are thinking as they answer, or, if that feels awkward, go through each question afterwards. These prompts work for a short cognitive interview:

• What do you think this question is asking?
• How did you choose your answer?
• Was there an answer you wanted that was missing?
• Was any word unclear, or did any two answers both fit?
• Was there a question you would rather not answer?

Two testers describing the same question differently is the clearest sign of a problem. Common causes are a double barrelled question asking two things at once and a leading question that suggests its own answer.

Step 4: fix three kinds of problem

Wording. “How satisfied are you with our service and prices?” asks two things. Split it: one question on service, one on prices.

Options. “How often do you visit? Weekly, monthly, sometimes” mixes a frequency with a vague word. Use consistent bands: weekly, monthly, a few times a year, rarely or never.

Flow. A jump rule meant to skip a section for people who answered no can send them past a question they needed. Walk every path a tester could take.

Step 5: retest the fixed version

A fix can create a new problem, so run one or two fresh testers on the changed questions before launch. Then freeze the wording. Changing a question after responses have arrived splits its results into before and after.

Pilot, pilot study and the rest of the build

A survey pilot tests the questions; a pilot study tests a whole research procedure, of which the questionnaire is one part. For writing the questions in the first place, see how to create a survey and how to write survey questions. The employee pulse survey and the course evaluation form are short enough to pilot in a lunch break.

What a pilot finds, in practice

In a small business survey, the problems a pilot finds are rarely subtle. A question uses an internal name nobody outside knows, such as a product code or a team name. An answer list has no option for the tester, such as a cafe survey asking which day they visit with no option for “it varies”. Two options overlap, such as age bands of 18 to 25 and 25 to 35. A rating scale runs the opposite way from the one before it. A required question asks for something the tester does not have to hand, such as an order number.

Each of those produces bad data without anyone noticing, because the respondent still picks something. The tester who hesitates and says “I wasn’t sure which one to pick” is giving you the most valuable comment of the afternoon.

Keep a simple log as you go: the question number, what the tester did or said, and the change you made. When the second round of testers moves through without pausing on the same questions, the survey is ready. If a change was large, such as rewriting a whole block, test that block again rather than trusting the fix.

In the product

A form can be shared as a link to testers before the real launch, and changes are made in the builder and published again. Publishing writes a new version and keeps the same link, and an earlier version stays reachable at its own version address. Conversational forms can carry jump rules, which are worth walking through on every path during the pilot. The update without changing the link page explains versions, and the guide to publishing updates without breaking your live link shows the steps.