What is a ranking question?

The cognitive cost is the real constraint

Ranking five items means holding five things in mind and comparing them against each other, which is roughly ten mental comparisons. Ranking ten items means forty five. That is why ranking questions with long lists produce a reliable pattern: the first two or three positions are considered, and the rest are filled in almost at random to get to the next question. If your analysis treats position eight as meaningful, it is reading noise. Keep ranking lists to five items, six at the outside, and if you have fifteen candidates, use a different method. Asking people to pick their top three from a list, without ordering them, is usually the better instrument, since it captures the part of the ranking people actually thought about and discards the part they did not.

What the numbers can and cannot tell you

Mean rank is the usual summary and it is easy to misread, because it hides disagreement. Two items can share a mean rank of three when one was ranked third by everybody and the other was ranked first by half the sample and last by the other half. The second item is the interesting one and the average erases it. Report the distribution of positions per item alongside the mean, or count how often each item appeared first. What ranking can never tell you is whether anybody liked any of it. A list of five options ranked cleanly says nothing about whether the winner was wanted, which is why a rating question or an open follow up usually belongs alongside.

Wording that prevents most of the confusion

Say explicitly what one means. Rank these from most important to least important, where one is the most important, removes the single largest source of reversed answers. Say whether every item must be ranked or whether a partial ranking is acceptable. Avoid a mixed list where some items are concrete and others abstract, since people cannot compare a four day week against a culture of trust. Randomising the presentation order across respondents is the standard defence against position effects in ranking, and where that is not available, choose a neutral order such as alphabetical rather than one that reflects the organisation's own preference.

Better alternatives for long lists

Three work well. Top three selection, which captures priorities at a fraction of the effort. A constant sum, where people distribute a hundred points across the options, which reveals intensity as well as order but takes more explaining. And a maximum difference approach, where people see small subsets and pick the best and worst in each, which produces a robust ranking from short easy questions at the cost of needing several of them. For most everyday surveys the top three question is the honest choice, and it has the further advantage that the results are easy to explain to somebody who was not involved in designing the survey.

Asking it without a drag and drop field

There is no ranking component, so a ranking question is built one of two ways. The first is a dropdown per rank: a dropdown asking which item is first, another asking which is second, and so on, each listing the same items. It is clear and it exports cleanly, but nothing stops somebody choosing the same item twice, so the wording has to ask them not to and the analysis has to check. The second is a matrix, with items as rows and rank positions as columns using radio style, which allows one answer per row. The matrix exports as one column with each statement and its answer joined as Statement: Answer.

Reporting a ranking so people believe it

A ranking result is easy to present badly. A bar chart of mean ranks with no sample size and no spread invites the reader to treat a gap of a tenth of a position as a finding, and somebody will build a decision on it. Three things make the report honest. Show how many people answered, because ranking questions attract more partial completion than most. Show the full distribution per item, usually as a stacked bar of how often each item landed in each position, so a divisive item is visible rather than averaged away. And state what was not asked: a ranking says nothing about whether any option was actually wanted, so if the open follow up produced a repeated request for something not on the list, that belongs next to the chart rather than in an appendix. Order the chart by the summary you are leading with, and say which one it is.

Questions people ask