Compared to What?
… is probably one of the most important questions there is. “Compared to what?” is the question that should be answered everytime it is about data, be it presented in graphs, in newspapers, on websites, or when just talked about. Yet it is rarely answered, rarely asked, and people end up knowing less or even wrong information.
The first time I heard about “compared to what?” was when reading Tufte who writes about it in context of information design, and ever since I am reminded of it when reading news; I finally felt the necessity to write about the idea when coming across Cory Doctorow’s article on rare events that kind of reveals a misconception as people probably aren’t that bad in understanding rare incidents, they rather don’t get an answer to above’s question, and they don’t ask either, “compared to what?”
“Compared to What?” in Action
So let’s go through a couple of examples. For instance, you read about (an example that I like to use in discussions about the concept) 100 people dying in car accidents each month in some city. Horrible. You probably don’t want to get into a car anyway anymore, unless you ask for additional data. And asking how many people actually live in the city in question might be a good start, and what impact does it have when the answer is 2.000, talking a small village, or 4.000.000 residents, talking a large city? How do you judge these 100 car accidents in each case? “Compared to what” can now mean the difference between staying in your apartment all day as it’s just too dangerous outside, or it might mean that you sue the journalist because of making all people nervous without a reason, as the large city is probably one of the safest places in the world when it comes to traffic accidents.
Figure: Data comparison. Copyright Kaiser Fung.
Another example is your company’s performance. Let’s say that last year, your sales dropped by 10 %. Bad. “Compared to what?” might make you have a closer look at your competitors’ performance. Does your 10 % decrease change meaning when all your competitors lost 25 % on average, or when sales generally went up 10 %? Of course; your company might perform very well despite the observed development, or you’re doing really bad. So again, panicking because of a single number that lacks any relation to other metrics might not be the best choice.
Yet another example might refer to terrorist acts. People hear about suicide attacks and casualties but never dare to ask how rare they basically are, in what context they happen, and how unlikely it is to be affected by such a attack even when walking Baghdad – it might never happen, and yet governments cut civil rights wherever they can in order to create the illusion of absolute safety.
Conclusions
However, the examples above must not deflect from two things:
-
There is no obligation to ask “compared to what?” only once, as you can and sometimes should go for even more data in order to be able to act on them.
-
“Compared to what?” should be answered anyway, so one might consider e.g. newspaper articles not providing data for comparison unprofessional. It is barely acceptable that news sources regularly fail in providing what is necessary to assess the data being presented.
So please make sure you answer “compared to what?” when revealing data, and never hesitate to ask “compared to what?” when being presented some data.
Read More
Enjoy the most popular posts, probably including:
Comments
-
On July 3, 2008, 16:14 CEST, Jonas said:
Great article.
It’s something people should be more critical about because it is often used for propaganda or for other political aims – like cutting the civil rights or letting the political achievements of the other parties seem worthless or not existent.
-
On July 3, 2008, 16:28 CEST, Duluoz said:
Great article Jens! I have theory as to why many people, at least in American culture, are hesitant to ask this very important question as well as similar questions of “Why”. I don’t know if they show these cartoons, etc., in European countries, but have you seen a cartoon where there is a child of around 3 or 4 years old who is asking “Why” over and over to everything a parent is saying? The parent then gets more and more aggravated and eventually explodes in frustration telling the child to stop asking questions? Take that, along with every teacher you ever had giving you the “my God is that child ever stupid for asking that”, and you’ve got a pattern of don’t ask, you should take it at face value, behavior/mentality.
-
On July 3, 2008, 23:55 CEST, Neovov said:
Yeah, great article, but compared to what?
I asked myself this question before reading your (nice) post and I’m pleased to read about it, especially here!
-
On July 4, 2008, 11:34 CEST, Steve Baty said:
Jens, I’m happy to see you raise this issue. It’s increasingly common to see data presented in a way that completely fails to provide the reader with the ability to critically assess the importance and severity of the issue. Meaningless statistics are everywhere, and don’t just arise in the absence of comparative data.
We need, as consumers of data (in the media, in reports, in research) to critically question the meaning of that data: does it provide illumination or, like the drunk & the lightpost, serve only for support.
You see examples of this when, for example, one metric is compared to another, unrelated data point. An example I like was sent to me from a charity: it compared the mean for one group, with the extreme upper value of a second group (hoping you would not notice the sleight of hand).
You also see it when two averages are presented without any mention of standard error, sample size, or variance. We’re seeing it at the moment with the discussions around the price of oil - with no mention of the effect the decline in the value of the US dollar is having on the benchmark prices. It makes for better headlines if we leave out that ‘complicating’ factor. Which is not to deny the rise in the real price of oil, but a discerning reader should be made aware of these issues.
As a society our media have trained us to look for ‘headline’ numbers - the single number that somehow captures everything we ‘need to know’, and to trust that there’s nothing left unsaid. And we automatically treat these figures as authoritative, although they’re often patently misleading.
We need to fight this trend, and I’m pleased to see you raise the topic for debate.
Steve
-
On July 4, 2008, 16:09 CEST, Rob said:
Great article Jens. Working in a corporate environment data interprets are often looked at in isolation. Time to ask: “Compared to What?”
-
On July 11, 2008, 13:20 CEST, Jens Meiert said:
Jonas, thank you; especially political implications leave “disquieting” feelings, yes.
David, interesting point, though I do not know this exact comic. Questioning data/information should not be discouraged, no, not at all.
Nicolas, cheers
Steve, these are great points, thank you! I am glad to see that there is much more awareness of this problem than I thought there was, actually.
Rob, thank you, and I do hope people ask this question more often (like I hope people stop blaming themselves for struggling with bad design, by the way; felt reminded of that).
-
On August 1, 2008, 23:28 CEST, lazar said:
It’s also important to ask “How?” and “Why?”. Sometimes even “Where?” and “When”?
How do 100 people die in car accidents – that is, how do those accidents happen? Maybe it’s because there are gang fights, maybe because people in that city like to drive some unsafe car model. Or maybe DUI limit is lest strict.
But yes, asking questions is always important.
ps. i can see that you posted quite a few interesting entires since the last time i visited.
-
On August 2, 2008, 14:09 CEST, Jens Meiert said:
Lazar, cheers! And you’re certainly right that some additional questions don’t hurt either in order to understand information, unless the respective source already answers them.
