Ask mathematicians, and they’ll tell you that the numbers never lie, they’re always telling the truth. Then, ask an economist, and you’ll probably get a completely different answer: “torture the numbers and they’ll confess to anything!”1 So where’s the truth?
As business leaders, we often use data to find out what is really going on. We are using numbers to understand what is happening and what we should do. We like to make informed decisions (at least, as much informed as we can). Yet, how can we be sure about what the data is telling us?
Getting the data is one thing. Making sense of what the data is telling us is completely different. We have to be very careful in the interpretation, because if we miss the true ‘message’, we will probably end up with the wrong decision. And it’s not the data we should blame…
A very powerful example about how data can be misinterpreted even by smart people, comes from a story from World War II. In 1943, US Army formed the Statistical Research Group (SRG), a classified programme. SRG was assembled by statisticians, and its main task was to develop statistical algorithms for improving the effectiveness of the US Air Force: optimal curves the pilots should follow to keep an enemy plane in gun sight for longer period, protocols for bombing enemy targets with higher precision, that kind of stuff.
One of the challenges brought to SRG was to decide on which part of a plane should be more armored than the others. It had to be the most crucial/vulnerable part only, because if excess armour was placed everywhere, that armour would make the plane much heavier, and heavier planes would have two big problems: they would be less maneuverable and they would use more fuel. On the other hand, planes could not stay without excess armoring in their most vulnerable part, since the enemy planes would shoot them down more easily. So the challenge for SRG was to find the most vulnerable part, i.e. that part where ectra armour would be placed.
SRG asked for data. The Air Force maintenance crew had collected a particular set of data, which they thought it would be useful: the number of bullet holes on different parts of the planes, when they were coming back from air combats. So they provided SRG with the following data set:
Location (on the plane)
Bullet holes per square meter
Engine
11.9
Fuselage
18.6
Fuel tank
16.7
Rest of the plane (nose, wings, etc.)
19.4
The above data shows that those planes who returned from air combats had on average around 19 bullet holes/m2 in their fuselage and their wings, around 17 in their fuel tanks and less than 12 in their engines. Therefore at first glance, data shows that the fuselage was being hit more frequently than (say) the fuel tank. Would that mean that the extra armour should rather go to the fuselage, or the wings, than to the fuel tank, or the engine? The maintenance crew believed that the answer was definitely “yes”, but they needed confirmation from SRG scientists. The final answer from SRG however was completely different…
The “Yes” answer would have been a disastrous decision! To understand why, let’s put the data into context: where does this data come from? It doesn’t come from the planes which go to combats, it comes from the ones which return. And –unfortunately- not all of the planes were coming back; some planes had been shot down… Therefore the above data set was not representative for all the planes who were fighting.
Secondly, the statisticians at SRG asked themselves a very insightful question: why the damages (i.e. bullet holes) were not spread equally all over a plane? Unless the enemy had a strange preference to shoot the fuselage, or the wings (which is very unlikely), the damage should have been equally spread across the entire body of the planes! But the data was telling the opposite. So where were the ‘missing’ holes? And all of a sudden, the answer became obvious: the missing holes were on the missing planes!
That insight made everyone to see the same data from a quite different perspective: the reason planes were coming back with fewer holes in the engine, is that if they were hit in the engine, they were less likely to come back! On the contrary, if planes were hit in the fuselage, they were more likely to come back. Hit in the engine was much more dangerous. And that’s where the extra armour had to be placed! That was the final recommendation form SRG, which saved many planes until the end of the War.
Indeed, that data was telling the truth. Numbers are always telling the truth! But we must know which truth exactly they are telling. We must know the limitations, and ask insightful questions before we reach to a conclusion. We must know how to read the data!
1 Quote attributed to Ronald Coase, British Economist & writer
