Forms · Glossary

What is non-response bias?

Non-response bias is the error that appears when the people who do not answer a survey differ from those who do in ways that affect the results. It is not the same as a low response rate: a survey with few replies can be unbiased, and one with many can be skewed if the missing people share a view.

It is the quiet failure behind many confident survey results, because nothing in the collected data announces who is missing. The only defence is to think about who stayed silent before reading what everyone else said.

Nuwan Madhusanka · Co-founder

5 min read · Published 13 September 2026

Signs of non-response bias and what helps
Sign	What it suggests	What helps
Respondents skew on a trait you can check, such as team or tenure	The people who answered no longer mirror the population	Weighting to known totals, or a reminder aimed at the thin group
Late responders answer differently from early ones	Reluctant people hold different views, and non-responders may differ further	Reminders, then comparing the answer waves
The topic matters much more to some people than others	Enthusiasts and critics are overrepresented	Invitation wording that does not signal a side
Results sit far from an outside benchmark	Taking part is linked to the answer	Validating key figures against records
A key group barely responded	Its view is guessed rather than measured	Following up by another channel, or reporting the group separately
Sensitive questions are skipped far more than others	Selective skipping inside the survey	Optional sensitive items, with their own answer counts reported

How the bias arises

Two things have to be true at once. Some people must fail to respond, and the chance of responding must be connected to the answer. If the people who ignore a staff survey hold the same views as those who complete it, a low rate costs precision and nothing else. If the most disengaged employees are also the least likely to answer an engagement survey, the result looks healthier than the organisation is, and no amount of extra responses from the engaged group will correct it. Pew Research Center's 2017 study of telephone polls with response rates of around 9 percent shows both cases in one study. On party identification, low response polls stayed within 1.4 and 1.6 points of a high response benchmark. On volunteering, the telephone estimate was 46 percent against 8 percent in the government's Current Population Survey.

A high response rate is not a cure

The instinct is to chase the rate, and a higher rate does shrink the room for bias. It does not remove it. AAPOR notes that studies comparing survey estimates with census benchmarks have questioned the assumed link between response rates and quality, and that some of the least biased results came from surveys with less than optimal rates. Pew's 2019 review found little relationship between response rates and accuracy across its studies, while warning that a low rate signals higher risk. The practical reading is that pushing a rate from 30 to 40 percent with aggressive reminders can bring in rushed, reluctant answers, and the bias you were worried about may simply move rather than shrink. Effort is better spent finding out who is missing and whether it matters for the questions being asked.

Checking for it

Three checks are within reach of any survey. First, compare the respondents with what is already known about the population: team sizes, regions, customer types, account age. A group that makes up a third of customers and a tenth of responses is a warning. Second, compare early and late respondents. People who answered only after the second reminder are a rough stand in for people who never answered, so if their answers lean one way, the missing group probably leans further. Third, where it matters enough, follow up a small random selection of non-responders by a different route, such as a short phone call, and compare their answers on the two or three questions that drive the decision. None of these proves the absence of bias, but each one can reveal its presence cheaply.

Reducing it before and after

Before the survey, the aim is to make taking part less dependent on caring about the topic. Dillman's Tailored Design Method uses several planned contacts, a clear reason why this person was asked, a realistic length and an easy first question. After the survey, weighting can rebalance respondents on traits with known population totals. Its limits are real: Pew's 2018 comparison of weighting methods found that weighting on demographics alone only minimally reduced bias and in some cases made it worse, and that the samples that improved most were adjusted on variables closer to the topic being measured. Weighting cannot recover a view that almost nobody in a group expressed.

What a form can and cannot tell you

A stored response holds the answers, the time submitted and the version of the form, and no IP address, browser details or account. On an anonymous survey that protects respondents and also means the form itself cannot say who is missing. To check, ask the traits you can compare against records, such as team or length of service, as dropdowns so they export as their own columns, and use the Submitted At column in the CSV to separate early responses from those that followed a reminder. A list of who has not responded has to come from your own invitation records.

Questions people ask

What is the difference between unit and item non-response?

Unit non-response is a person who does not take part at all. Item non-response is a person who takes part but skips particular questions. Both can bias results. Item non-response is easier to see, because the gaps are in your data, and it tends to cluster on sensitive or difficult questions such as income, health and criticism of a manager.

Is non-response bias the same as self selection bias?

They are close relatives. Non-response bias applies when a defined sample was invited and some declined. Self selection bias applies when there was no defined sample, such as a public link, and people decided for themselves to take part. Self selection is usually worse, because there is not even a list to compare respondents against.

Do incentives reduce non-response bias?

They can, when they persuade people with little interest in the topic to take part, which spreads participation more evenly. They can also add a different skew by appealing most to people who value the reward. A small incentive offered to everyone is the safer version, and it is worth checking whether it changed who answered, not just how many.

Does making every question required prevent it?

No. Requiring answers turns some item non-response into unit non-response, since people who will not answer one question abandon the whole survey, and it can produce made up answers from people who just want to finish. Make the key questions required, leave sensitive ones optional, and offer a prefer not to say option where it fits.

How should non-response be reported?

State the number invited, the number who responded, the dates, and how the respondents compare with the population on traits you could check. Say which groups are thin and whether results were weighted. A short honest paragraph lets readers judge the risk themselves, which is far better than a footnote nobody reads.

Is there a response rate that rules it out?

Only a complete census where everybody answers. Short of that, the risk depends on whether the missing people differ on the questions asked, not on a threshold. A 90 percent rate with the most unhappy tenth missing can mislead more than a 40 percent rate spread evenly across the population.

Make one with forms

The button opens the generator with this use case already described. Change the wording to match your own.

Create a form with OneCraft

Sources

Written and checked by the OneCraft team. Last checked 13 September 2026.