Commentary & Analysis

Online Polls: The Internet Is Doing Them Wrong

You see them on the internet all the time. No, not cat videos. I’m talking about online polls. And they look a lot like the one above from Entertainment Weekly.

So what’s so special about online polls, you ask? Answer: We’re doing them wrong.

Take a look at the Entertainment Weekly poll. The first thing we see is that it uses radio buttons. This means the respondent is required to choose only one selection. Choose another option and you switch the selection.

What could be wrong with limiting people to select one movie here? When you force people to choose only one, support artificially divides between options, particularly among similar options. The product is a polling result that can be practically meaningless.

Let’s play this poll out. I like comedies, so there are a number of good options. And although it’s not a stereotypical Christmas movie, I also like Die Hard (unlike the newer Die Hard atrocities). But if I was forced to pick one option, I’d have to go with Scrooged. Hey, it’s Bill Murray.

Now for the results: A Christmas Story wins, with Love Actually just behind. My personal favorite, Scrooged, came in … good grief … last with just over 3%.

Perhaps it makes sense that A Christmas Story won, although It’s a Wonderful Life beats it by over half a point on Internet Movie Database (IMDB). But why did Love Actuallycome so close to winning? After all, Love Actually’s IMDB score is a full point lower than It’s a Wonderful Life, which it beat.

Aside from Die Hard, Love Actually, a romance comedy, is the most dissimilar to the other options. Any support it got was less likely to be divided. Conversely, there were three classic movies that could divide support, and there were several other traditional comedies (naturally explaining why Bill Murray’s film got such little support!).

This artificial division of support among options—what we call vote splitting—is a trademark of choose-one voting and polling. It can flat out give you the wrong answer, and it reliably distorts the results of the remaining options, especially when the options are remotely similar.

There is an exception when it’s okay to force one selection in a poll. You should limit the respondent’s selection when the options are mutually exclusive, i.e. when it’s only feasibly possible for the respondent to have one answer. An example might be, “How many times have you eaten out this month?” If the options are 0-2, 3-8, and 9+, then it makes no sense for someone to have eaten out both between both 0-2 as well as 9+ times. It’s okay to limit the respondent to selecting one option here. Polls with only two options are also an acceptable use of choose-one polling.

I’m not picking on Entertainment Weekly here with their poll. This style of polling is ubiquitous on the Internet. You’ll unfortunately find nearly everyone using it from Mashable to even Lifehacker.

Now how do we avoid this mistake? Many polling services allow you to pick multiple options, using checkboxes instead of radio buttons. This switch from a single-selection to an unlimited selection technically switches us from a plurality voting method to approval voting—an enormous improvement.

When we do this, we wind up with what we see to the right: a poll with checkboxes. Permitting the respondent to choose multiple options sidesteps the vote-splitting issue. Like similar options? Choose each one. With this approach, we can be more confident in the winner, and alternatives that don’t top the leaderboard still get their fair share of support.

If you’re feeling extra feisty, you can take this to the next level and allow respondents to score their options. You can see an example of this below. Technically, this is called score voting. This permits maximum expression.

So please, internet, stop doing choose-one polls, and use approval polls! It hardly ever makes sense to limit respondents to one selection. Use those checkboxes or allow for ratings so we can get online polling results that actually make sense.