What is survey weighting?

Working through the example

Tidewater Hardware's customers are split evenly between online and in store buyers, but online buyers answered the survey far more readily, so 70 percent of responses came from them. Online buyers were also happier. Left alone, the survey reports 74 percent satisfied, a figure pulled upward by whichever group was keener to reply. Each weight is the group's share of the population divided by its share of the sample: 50 divided by 70 gives 0.714 for online responses, and 50 divided by 30 gives 1.667 for in store responses. Multiply each response by its weight and the two groups count as 200 each, matching the real split. Satisfaction then comes out at 70 percent, the simple average of 80 and 60. Four points is the difference between reporting a stable year and reporting a decline worth investigating, and it came entirely from who chose to answer.

The main methods

The example uses cell weighting, which works when a sample is split on one or two traits with known population shares. Real surveys usually need to balance several traits at once, and the cells quickly become too small. Raking, the most common method, handles this by adjusting the weights for one trait, then the next, and cycling until the sample matches the population on each trait separately, which needs only the overall share for each rather than every combination. Pew Research Center's 2018 comparison also describes matching, where each case in a target population sample is paired with a similar respondent and unmatched respondents are discarded, and propensity weighting, where respondents who resemble people rarely found in the survey receive larger weights. The best performing approaches in that study combined several of these steps.

The cost: a wider margin of error

Weighting corrects the balance of a sample by leaning harder on some responses, and leaning on fewer voices increases uncertainty. Pew notes that weighting makes the margin of error larger, which statisticians call the design effect. One common way to see it is the effective sample size: square the sum of the weights and divide by the sum of the squared weights. In the Tidewater example the 400 responses behave like 336 unweighted ones, so the margin at 95 percent confidence for a 50 percent result widens from about 4.9 to about 5.3 points. That is a small price for a four point correction. The price becomes steep when one group is badly underrepresented and each of its responses carries a weight of five or ten, because a handful of people then decides the headline.

What weighting cannot fix

Weighting can only balance the traits it is given, and only when reliable population figures for those traits exist. If online buyers who answered differ from online buyers who did not, no weight based on channel will notice. Pew's 2018 study found that weighting on core demographics alone only minimally reduced bias, and in some cases made it worse; the samples that improved most were adjusted on additional variables more closely related to what was being measured. Pew's 2019 review of telephone polling makes the same point from the other side: standard weighting sometimes corrects the link between who takes part and what they say, and sometimes, as with volunteering, it does not. Weighting also cannot recover a group that is almost absent. Ten responses from a group that makes up a third of the population carry weights so large that the result is mostly noise, and the honest report shows that group separately with its count.

Weighting form responses in a spreadsheet

The CSV export writes one column per field, with choice questions written out as the option labels people chose, so a single choice question such as online or in store arrives ready to use as a weighting variable. Ask that question as its own radio or dropdown field rather than a row in a rating grid, which would export joined with other statements. Take the population shares from records rather than guesses, add a weight column with a lookup on that label, and calculate weighted percentages by summing weights.

Questions people ask