Compared to What?
Post from July 3, 2008 (↻ June 12, 2021), filed under Art and Design.
…is probably one of the most important questions there is. “Compared to what?” is the question that should be answered every time it is about data, be it through charts, in newspapers, on websites, or in conversations. Yet it is rarely asked, rarely answered, and people end up with less or even false information.
The first time I heard about “compared to what?” was when reading Tufte who writes about it in the context of information design. Ever since I am reminded of it when reading news. I felt the need to write about the idea when coming across Cory Doctorow’s article on rare events that may reveal a misconception as people probably aren’t that bad in understanding rare incidents, they rather don’t get an answer to the question above: “Compared to what?”
“Compared to What?” in Action
Let’s go through a couple of examples. For instance, you read about (an example that I like to use in discussions about the concept) 100 people dying in car accidents each month in some city. Horrible. You probably don’t want to enter a car anymore—unless you ask for additional data. And asking how many people actually live in the city in question might be a good start. What does it mean when the answer is 2,000, talking a small village, or 4,000,000 residents, talking a large city? How do you judge these 100 car accidents in each case? “Compared to what” is the difference between staying in your apartment all day, or suing the journalist because of freaking everybody out without a reason.
Figure: Data comparison. (Copyright Kaiser Fung.)
Another example is your company’s performance. Let’s say that last year, your sales dropped by 10%. Bad. “Compared to what?” might make you have a closer look at your competitors’ performance. Does your 10% decrease change meaning when all your competitors lost 25% on average, or when sales generally went up 10%? Of course; your company might perform very well despite the observed development, or really bad. So again, panicking because of a single number that lacks any relation to other metrics is not a good choice.
Another example involves, dare I say it, terrorist acts. People hear about suicide attacks and casualties but never ask how rarely they occur, in what context they happen, and how unlikely it is to be affected by an attack even in Baghdad—it might never happen. Yet governments cut civil rights where they can, addressing the illusion of a threat through an illusion of safety.
Two more things:
“Compared to what?” should always be answered. One might, by default, consider newspaper articles unprofessional that don’t provide data for comparison. It is not acceptable that news sources fail to provide what is necessary to assess the data being presented.
There is no obligation to ask “compared to what?” only once. You can and sometimes should go for more information in order to be able to learn and act on it.
The simple conclusion: Don’t hesitate to ask “compared to what?” when being presented with data, and make sure to answer “compared to what?” when displaying data.
I’m Jens Oliver Meiert, and I’m an engineering manager and author. I’ve worked as a technical lead for Google, I’m close to the W3C and the WHATWG, and I write and review books for O’Reilly. Other than that, I love trying things, sometimes including philosophy, art, and adventure. Here on meiert.com I share some of my views and experiences.
If you have questions or suggestions about what I write, please leave a comment (if available) or a message.
On July 3, 2008, 16:14 CEST, Jonas said:
It’s something people should be more critical about because it is often used for propaganda or for other political aims – like cutting the civil rights or letting the political achievements of the other parties seem worthless or not existent.
On July 3, 2008, 16:28 CEST, Duluoz said:
Great article Jens! I have theory as to why many people, at least in American culture, are hesitant to ask this very important question as well as similar questions of “Why”. I don’t know if they show these cartoons, etc., in European countries, but have you seen a cartoon where there is a child of around 3 or 4 years old who is asking “Why” over and over to everything a parent is saying? The parent then gets more and more aggravated and eventually explodes in frustration telling the child to stop asking questions? Take that, along with every teacher you ever had giving you the “my God is that child ever stupid for asking that”, and you’ve got a pattern of don’t ask, you should take it at face value, behavior/mentality.
Yeah, great article, but compared to what? 😉
I asked myself this question before reading your (nice) post and I’m pleased to read about it, especially here!
On July 4, 2008, 11:34 CEST, Steve Baty said:
Jens, I’m happy to see you raise this issue. It’s increasingly common to see data presented in a way that completely fails to provide the reader with the ability to critically assess the importance and severity of the issue. Meaningless statistics are everywhere, and don’t just arise in the absence of comparative data.
We need, as consumers of data (in the media, in reports, in research) to critically question the meaning of that data: does it provide illumination or, like the drunk & the lightpost, serve only for support.
You see examples of this when, for example, one metric is compared to another, unrelated data point. An example I like was sent to me from a charity: it compared the mean for one group, with the extreme upper value of a second group (hoping you would not notice the sleight of hand).
You also see it when two averages are presented without any mention of standard error, sample size, or variance. We’re seeing it at the moment with the discussions around the price of oil - with no mention of the effect the decline in the value of the US dollar is having on the benchmark prices. It makes for better headlines if we leave out that ‘complicating’ factor. Which is not to deny the rise in the real price of oil, but a discerning reader should be made aware of these issues.
As a society our media have trained us to look for ‘headline’ numbers - the single number that somehow captures everything we ‘need to know’, and to trust that there’s nothing left unsaid. And we automatically treat these figures as authoritative, although they’re often patently misleading.
We need to fight this trend, and I’m pleased to see you raise the topic for debate.
On July 4, 2008, 16:09 CEST, Rob said:
Great article Jens. Working in a corporate environment data interprets are often looked at in isolation. Time to ask: “Compared to What?”
It’s also important to ask “How?” and “Why?”. Sometimes even “Where?” and “When”?
How do 100 people die in car accidents—that is, how do those accidents happen? Maybe it’s because there are gang fights, maybe because people in that city like to drive some unsafe car model. Or maybe DUI limit is lest strict.
But yes, asking questions is always important.
ps. i can see that you posted quite a few interesting entires since the last time i visited.
Have a look at the most popular posts, possibly including:
Looking for a way to comment? Comments have been disabled, unfortunately.
Perhaps my most comprehensive book: The Web Development Glossary (2020). With explanations and definitions for literally thousands of terms from Web Development and related fields, building on Wikipedia as well as the MDN Web Docs. Available at Apple Books, Kobo, Google Play Books, and Leanpub.