Forms · Glossary
What is an attention check question?
An attention check question is an item placed in a survey to test whether respondents are reading carefully, such as an instruction to select a particular answer. People who fail it may be skimming, answering at random or not human at all, so researchers use these items to flag low quality responses before analysing the data.
Attention checks are the survey equivalent of a spot check. Used sparingly and fairly, they improve data quality; overused, they insult the people answering honestly.
Nuwan Madhusanka · Co-founder
3 min read · Published
| Type | Example | What it catches |
|---|---|---|
| Instructed response item | For this statement, please select Disagree | Straight lining and skimming |
| Bogus or impossible item | I have visited the moon | Random clicking |
| Consistency check | The same question asked twice in different words | Careless or contradictory answers |
| Speed check | Completion time far below the realistic minimum | Speeding through without reading |
How attention checks work
The researcher adds an item whose correct answer is obvious to anyone reading. In an instructed response item, the question tells the respondent which option to pick. In a bogus item, the statement is impossible, so agreeing with it signals inattention. A 2009 paper by Oppenheimer, Meyvis and Davidenko in the Journal of Experimental Social Psychology introduced the instructional manipulation check, a longer instruction buried in a question, and found that excluding people who failed it reduced noise in the results.
Why they matter more online
Online surveys, especially those using opt-in panels or incentives, attract some respondents who click through as fast as possible and, increasingly, automated responses. Pew Research Center's 2020 analysis found about 4% to 7% bogus respondents in opt-in online samples, against about 1% in panels recruited from residential addresses, and that they tended to pick positive answers. It also found that 84% of the bogus respondents passed the attention check question used, so a check on its own is a weak filter.
Using them fairly
Keep checks few, one or two in a typical survey, and make them clearly worded so an attentive person cannot fail by misreading. Decide before collecting data what failing means: removal, or a flag for review. Avoid trick questions that test reading speed rather than attention. Tell participants in research studies that checks are included, if your ethics approval requires it, and never penalise respondents in ways they did not agree to.
Common mistakes
Checks that are ambiguous, so attentive people fail. Removing everyone who fails one check, which can bias the sample towards particular groups. Placing a check as the first question, before people have settled in. Using attention checks on a short customer feedback form, where they mostly annoy. And relying on them alone, when speed and pattern checks catch different problems.
Checks in a OneCraft form
A form built in OneCraft can include an instructed item as an ordinary choice question, and responses export to CSV with option labels, one column per field, so failed checks can be filtered in a spreadsheet. The builder does not score answers or remove respondents automatically. In a conversational form, a jump rule can route people on an answer, but no form logic marks a response as failed.
Questions people ask
How many attention checks should a survey have?
Usually one or two in a survey of fifteen to thirty questions. More than that starts to feel like a test and can irritate careful respondents. Short customer surveys of under ten questions rarely need any, because the effort of skimming them is not much less than answering properly.
Should I remove everyone who fails an attention check?
Not automatically. Decide your rule before collecting data and consider how strict to be. Failing one easy check alongside other warning signs, such as very fast completion, is stronger evidence than a single miss. Removing too many can change who is left in the sample.
Are attention checks the same as trap questions?
The terms overlap. Trap question usually means a check designed to catch inattention, such as an instructed item. Some people use it for questions meant to catch dishonesty, which is a different problem. Clear instructed items are fairer than tricks that rely on misdirection.
Do attention checks catch bots?
Not reliably. In Pew Research Center's 2020 study, 84% of bogus respondents passed the attention check and 87% passed the speeding check. Combine checks with duplicate detection, a review of open text answers and, where possible, a better recruited sample.
Can attention checks affect how people answer the rest of the survey?
Possibly. Some studies suggest that a check can make people more careful afterwards, which is partly the point. Badly worded checks can also make respondents feel distrusted. Keep them neutral in tone and place them away from the most sensitive questions.
Make one with forms
The button opens the generator with this use case already described. Change the wording to match your own.
Create a form with OneCraftRelated questions
- What is survey fatigue?Survey fatigue is the drop in response quality and completion as a survey gets longer or arrives too often. How length affects drop off and what to cut.
- What is acquiescence bias?Acquiescence bias is the tendency to agree with a statement regardless of its content. Why agree and disagree scales invite it, and what to use instead.
- What is a forced choice question?What is a forced choice question: one that makes respondents pick an answer with no neutral or skip option, the three ways the term is used, and when it helps.
Step by step in the builder: View, export and manage your responses.
Written and checked by the OneCraft team. Last checked .