What is a representative sample?

Representative on which traits

No sample matches a population on everything, so the useful question is which traits drive the answers you are measuring. A survey about public transport needs the right mix of people who live near train lines; a survey about a software product needs the right mix of heavy and occasional users. Matching on age and gender while missing the trait that matters produces a sample that looks balanced in the methods section and misleads in the results. Pew Research Center showed how large the gap can be. In 2016 it compared nine online samples from eight vendors against 20 benchmarks from high quality government sources. Average estimated bias on those benchmarks was 15.1 points for Hispanic adults and 11.3 points for Black adults, and every sample overrepresented politically and civically engaged people.

Probability and non-probability samples

Sampling methods split into two families. In a probability sample, as Qualtrics's guide puts it, each member of the population has a known, non-zero chance of being selected, which is what allows a margin of error to be calculated and a result to be generalised. Simple random, stratified, systematic and cluster sampling all belong here. In a non-probability sample the researcher, or the respondents themselves, decide who takes part on grounds such as convenience or availability, and the same guide warns these are unlikely to produce a representative sample. Non-probability samples are cheaper and faster, and for early exploration they are often good enough. Quota sampling tries to bridge the gap by filling set numbers per group, which fixes the proportions you set quotas on and leaves every other difference untouched. The vendor that performed best in Pew's comparison relied on an elaborate set of adjustments, not luck.

The sampling frame comes first

Before any method applies there has to be a list, or some other route, that reaches the population. That list is the sampling frame, and a sample can only represent the people on it. A customer database that holds only people who created accounts misses everyone who paid in cash. A staff directory can miss contractors. Pew's American Trends Panel shows the effort a strong frame takes: since 2018 it has recruited by selecting households from the U.S. Postal Service's master list of residential addresses, it adds new members each year because some stop responding, and it retires members from groups that have become overrepresented. For an ordinary business survey the lesson is smaller but the same. Write down who is on the list, who is not, and whether the missing people might answer differently.

Checking a sample you already have

Once responses are in, compare the sample's makeup with known facts about the population. A staff survey can be checked against headcount by team; a customer survey against the share of customers by region or product. Where the sample is off on a trait that matters, weighting can rebalance it, provided the population figures are reliable and no group is almost absent. Where a key group barely responded, weighting stretches a handful of answers too far, and the better report shows that group's count and says its result is uncertain. It also helps to compare early and late respondents, since the people who needed reminders are often closer to the people who never answered at all.

Keeping a form inside its frame

A public form link can be forwarded anywhere, which turns a careful random selection into a convenience sample one share at a time. When the frame is a known group, the login required access type limits the form to an allowed list of addresses, or to whole domains written like @company.com. That check happens when the form opens; the account is not saved with the answers, so it keeps outsiders out without identifying anyone. Ask the traits you will compare against population figures as single dropdown or radio questions, because each becomes its own column in the CSV export.

Questions people ask