Compared to What?

Published on July 3, 2008 (↻ November 19, 2022), filed under Design (RSS feed for all categories).

…is probably one of the most important questions there is. “Compared to what?” is the question that should be answered every time it is about data, be it through charts, in newspapers, on websites, or in conversations. Yet it is rarely asked, rarely answered, and people end up with less or even false information.

The first time I heard about “compared to what?” was when reading Tufte who writes about it in the context of information design. Ever since I am reminded of it when reading news. I wanted to write about it when coming across Cory Doctorow’s article on rare events—it may be based on a misconception as people probably aren’t that bad in understanding rare incidents, but don’t usually get an answer to the question: “Compared to what?”

“Compared to What?” in Action

Let’s go through a couple of examples.

For instance, you read about 100 people dying in car accidents each month in some city. Horrible. You probably don’t want to enter a car anymore—unless you ask for additional data. And asking how many people actually live in the city in question might be a good start. What does it mean when the answer is 2,000, talking a small village, or 4,000,000 residents, talking a large city? How do you judge these 100 car accidents in each case? “Compared to what” is the difference between staying in your apartment all day, or suing the journalist because of freaking everybody out for no reason.

Another example is your company’s performance. Let’s say that last year, your sales dropped by 10%. Bad. “Compared to what?” might make you have a closer look at your competitors’ performance. Does your 10% decrease change meaning when all your competitors lost 25% on average, or when sales generally went up 10%? Of course; your company might perform very well despite the observed development, or really bad. So again, panicking because of a single number that lacks a link to other metrics is not a good choice.

Another example involves, dare I say it, terrorist acts. We hear about suicide attacks and casualties, but we never ask how rarely they occur, in what context they happen, and how unlikely it is to be affected by an attack. Yet governments cut civil rights at an alarming rate, replacing the illusion of a threat by an illusion of safety.

Conclusions

Two more things:

1. “Compared to what?” should always be answered. One might, by default, consider newspaper articles unprofessional that don’t provide data for comparison. It’s not acceptable that news sources fail to provide what is necessary to assess the data being presented.

2. There is no obligation to ask “compared to what?” only once. You can and sometimes should go for more information in order to be able to learn and act on it.

The simple conclusion: Ask “compared to what?” when being presented with data, and make sure to answer “compared to what?” when displaying data.

About Me

I’m Jens (long: Jens Oliver Meiert), and I’m a frontend engineering leader and tech author/publisher. I’ve worked as a technical lead for companies like Google and as an engineering manager for companies like Miro, I’m a contributor to several web standards, and I write and review books for O’Reilly and Frontend Dogma.

I love trying things, not only in web development (and engineering management), but also in other areas like philosophy. Here on meiert.com I share some of my experiences and views. (Please be critical, interpret charitably, and give feedback.)

Comments (Closed)

1. On July 3, 2008, 16:14 CEST, Jonas said:

   Great article.

   It’s something people should be more critical about because it is often used for propaganda or for other political aims – like cutting the civil rights or letting the political achievements of the other parties seem worthless or not existent.

2. On July 3, 2008, 16:28 CEST, Duluoz said:

   Great article Jens! I have theory as to why many people, at least in American culture, are hesitant to ask this very important question as well as similar questions of “Why”. I don’t know if they show these cartoons, etc., in European countries, but have you seen a cartoon where there is a child of around 3 or 4 years old who is asking “Why” over and over to everything a parent is saying? The parent then gets more and more aggravated and eventually explodes in frustration telling the child to stop asking questions? Take that, along with every teacher you ever had giving you the “my God is that child ever stupid for asking that”, and you’ve got a pattern of don’t ask, you should take it at face value, behavior/mentality.

3. On July 3, 2008, 23:55 CEST, Neovov said:

   Yeah, great article, but compared to what? 😉

   I asked myself this question before reading your (nice) post and I’m pleased to read about it, especially here!

4. On July 4, 2008, 11:34 CEST, Steve Baty said:

   Jens, I’m happy to see you raise this issue. It’s increasingly common to see data presented in a way that completely fails to provide the reader with the ability to critically assess the importance and severity of the issue. Meaningless statistics are everywhere, and don’t just arise in the absence of comparative data.

   We need, as consumers of data (in the media, in reports, in research) to critically question the meaning of that data: does it provide illumination or, like the drunk & the lightpost, serve only for support.

   You see examples of this when, for example, one metric is compared to another, unrelated data point. An example I like was sent to me from a charity: it compared the mean for one group, with the extreme upper value of a second group (hoping you would not notice the sleight of hand).

   You also see it when two averages are presented without any mention of standard error, sample size, or variance. We’re seeing it at the moment with the discussions around the price of oil - with no mention of the effect the decline in the value of the US dollar is having on the benchmark prices. It makes for better headlines if we leave out that ‘complicating’ factor. Which is not to deny the rise in the real price of oil, but a discerning reader should be made aware of these issues.

   As a society our media have trained us to look for ‘headline’ numbers - the single number that somehow captures everything we ‘need to know’, and to trust that there’s nothing left unsaid. And we automatically treat these figures as authoritative, although they’re often patently misleading.

   We need to fight this trend, and I’m pleased to see you raise the topic for debate.

   Steve

5. On July 4, 2008, 16:09 CEST, Rob said:

   Great article Jens. Working in a corporate environment data interprets are often looked at in isolation. Time to ask: “Compared to What?”

6. On August 1, 2008, 23:28 CEST, lazar said:

   It’s also important to ask “How?” and “Why?”. Sometimes even “Where?” and “When”?

   How do 100 people die in car accidents—that is, how do those accidents happen? Maybe it’s because there are gang fights, maybe because people in that city like to drive some unsafe car model. Or maybe DUI limit is lest strict.

   But yes, asking questions is always important.

   ps. i can see that you posted quite a few interesting entires since the last time i visited.