Single select against multi select questions

The select all that apply undercount

Survey methodologists have documented this for decades: a checkbox list systematically collects fewer selections than the same items asked one at a time as yes or no. People satisfice, ticking until they feel they have answered enough and then moving on, and options further down the list get read less carefully than the ones at the top. The effect is large enough to change conclusions, so a checkbox list is a poor instrument when the count itself is the finding. If you need an accurate count of how many of twelve features somebody uses, asking twelve short yes or no questions produces better data, at the cost of a longer form. Where the checkbox list stays, keep it short, randomise or rotate the order where you can, and treat the resulting percentages as a floor rather than a measurement.

Making a single select genuinely single

A single select only works if the options are mutually exclusive and cover everything. Overlapping options are the commonest fault: a list offering email, social media and Instagram forces anybody who came from Instagram to guess which the writer meant. Coverage is the other half, and it is why so many lists end with an other entry. Note that dropdown, radio and checkbox fields have no working other option, so the honest build is a separate short text field placed after the choice question. Two more habits help. Put the options in a logical order rather than the order they occurred to you, and avoid a default selection, since a preselected option is chosen more often than it would otherwise be.

When multi select is the wrong instrument

Multi select cannot tell you which option matters most, only which ones were selected. If the decision depends on priority, a ranking or a top three question answers it and a checkbox list does not. It also struggles when the options are not equally weighted in people's minds, because ticking a box is the same action whether the item is central or incidental. A third problem is the empty answer: a respondent who ticks nothing may mean none of these or may have skipped the question, and unless the list carries an explicit none of the above option the export cannot distinguish them. Adding that option is cheap and it makes a real difference to how the data can be read.

Reading the results without overstating them

Report a single select as a share of respondents and say the base. Report a multi select as the share of respondents who selected each option, and say that people could choose more than one, because otherwise a reader will add the percentages and conclude the numbers are wrong. Avoid reporting a multi select as a share of mentions, which is technically possible and almost always misread. If you want to know how options travel together, a cross tabulation of the common pairs is more useful than a longer list of individual percentages, and it often reveals that two options were being treated as the same thing by respondents.

Building both in a form

Radio gives a single answer and checkbox gives several, and both can be laid out in one, two or three columns, which matters on a phone where a three column layout of long labels becomes unreadable. Use a dropdown instead of radio once the list passes about six or seven options. A matrix is the compact way to ask several statements against one shared scale, with radio style allowing one answer per row and checkbox style allowing several. In the export, radio, dropdown and checkbox fields all write the option labels rather than the stored values, one column per field.

Writing the option list

Most of the quality of a choice question sits in its options rather than its stem. Draft the list from what people actually said in an earlier open question rather than from a meeting, use their words rather than internal terminology, and check that no two options could describe the same situation. Keep the labels parallel in grammar and roughly equal in length, because an unusually long option attracts attention and an unusually specific one signals that it is the expected answer. Order matters too: put the options in a sequence the reader can predict, such as frequency from most to least or price from low to high, and keep any residual entry such as none of the above at the end. Finally, read the list as somebody outside the organisation would and ask which option you would pick if none of them quite fitted.

Questions people ask