Thinking rationally about big numbers

One difficulty you see in both politics and everyday life is in comprehending the meaning of numbers; things like "doubling your risk", "as many as 2 million British residents affected", or even "just 50p a day" get thrown around, and you are expected to somehow understand what these numbers mean to you, and whether they should or should not influence your behaviour. I'm not going to look at how these numbers get abused; I recommend Ben Goldacre's Bad Science and Mark Chu-Carroll's Good Math, Bad Math for that.

Instead, I'm going to discuss techniques you can use to help turn overwhelming numbers that you can't get a good grip on into numbers you can understand intuitively. What do I mean by intuitive? In this case, it means that you've got them into the range of numbers that you encounter every day and that you can reason about - this means small fractions (nothing much less than 0.05, or one twentieth), and small numbers (nothing more than a few dozen). It means bringing numbers that refer to big groups down to talking about the group of people you know personally. It means that you've got numbers where your instinctive feel for what a number means actually works properly.

So, let's start with an example; in their manifesto, the BNP claim that there are 300,000 to 500,000 third world immigrants to Britain every year. This is too big a number for me to comprehend instinctively, so my first reaction is to shrink it. The population of the UK is around 60 million, so the BNP's number is less than 1% of the population.

This, however, is still an awkward number to think about; I don't normally think about 1% of a person. So, let's look at this a different way; in a typical week, how many people do you interact with who you'll still recognise a week later (including things like the helpful checkout assistant at the supermarket, whose name you'll never remember, but who you'll recognise by sight)? I reckon there's under 50 of those in a week for me; so, the BNP statistic reduces down to half a person extra in my week who I might recognise in future. Even thinking about the number of people I interact with in a year who I'd still recognise later, I struggle to get to 100 people, which reduces the BNP statistic to "less than one person a year in the size of group I understand".

Once you reduce the scary big number to that, it's not so scary - you can now get into a more meaningful consideration of the number - is an extra person a year in the group you interact with something the country can absorb? Are they being absorbed into the general ebb and flow of British citizenry, or are you seeing Third World ghettos appear in your neighbourhood?

For a different example, imagine newspaper headlines telling you that getting up at 6am and going to bed at 6pm halves your chances of dying from a heart attack before you're 60. It sounds like you should change your habits to match the study; a 50% reduction is huge. However, on further reading, it's not nearly as impressive as it sounds: 94,000 people die of a heart attack in the UK each year, in a population of 60 million, which works out to around 0.16% of the population, or 1 person in every 600, each year. This includes people who die of heart attacks while over 60, which biases the analysis a little, but we'll continue with the known-faulty figure. A useful way to think of this is that in a typical group of (say) 10 people - yourself and your closest friends and family - you'd expect to get through 60 years before one of the group had a heart attack. If you make the group a bit bigger (say 30 people), you now only expect to get through 20 years before someone has a heart attack.

A halving of risk in this case doubles the time to expected heart attack, so our big group now has 40 years between heart attacks, not 20. And, of course, this figure is based on a known-bad assumption; we assumed that under-60s were as likely to have a heart attack as over-60s. However, we know have a more useful way to think about the headline claim - it means that a group of 30 people who follow the advice go from losing a member every 20 years due to heart attacks, to a member every 40 years. Thinking about it this way, you may decide you prefer not to worry about making drastic lifestyle changes.

Two examples, two different sets of reasoning; what's the link? In both cases, I started with a number I couldn't intuitively handle; either too large, or too abstract. I used external statistics to convert it from a raw figure to a probability or proportion I could apply to a group. Finally, I applied the probability to a group that I could picture in my head (all the people I deal with in a week, a group of my closest friends), instead of to an abstract group that I can't really think about. Once I'd done this, I had a number I could reason about rationally, and where my intuition about what it meant matched cold logic, instead of a scary number that I couldn't handle.

Science, pseudo-science and non-science

As someone who holds strong opinions, I'm often found arguing with people; because my education is scientific, the debate is often around a subject where "scientific evidence" is expected to carry the day. Most of the time, the claimed "scientific evidence" is nothing of the sort - often, it's nonsense using "sciency" words to seem right.

It's quite simple to tell science from non-science, and not that much harder to distinguish pseudo-science from science. If you're planning to argue using "scientific evidence", please make sure it's genuinely scientific before you start.

Non-scientific claims are based on authority, not on testable evidence. Examples include "my doctor tells me that this drug will suppress the symptoms of my hayfever", or "the Bible tells me that God loves everyone". Non-scientific claims don't have to be false; they can be a useful shortcut to avoid going over basic knowledge again and again, or you could be in a realm that science cannot speak on.

Scientific claims have two properties. Firstly, they are based on testable evidence; secondly, when the claim has been tested, it has been found to be "not false".

Pseudo-scientific claims are very similar to scientific claims. The difference is that where a scientific claim has been found to be not false, a pseudo-scientific claim has been found to be true.

The distinction between found not false and found to be true is subtle, and bears further examination; it's about the way in which you test the relationship between your claim and your evidence. When you find a claim "not false", you start with your claim, and think of a way to gather evidence that the claim is false. You then try to gather that evidence; the claim is found not false when your evidence does not contradict the claim.

In contrast, when you find a claim "true", you start with the claim, and try and gather as much evidence as you can of the claim being true. The claim is found to be true if the evidence does indeed agree with your claim.

This does not sound like a huge difference; however, in practice, people tend to find the evidence they set out to gather, unless that evidence genuinely does not exist. What's more, for many of the claims we wish to evaluate, there are many reasons why the claim could appear true.

For example, earlier in the year, I suffered from a bout of flu, lasting about 4 days in total; during that time, I took Tamiflu. Now, I could claim that Tamiflu cured my illness, and point to the fact that I had flu, I took a course of Tamiflu, and I was then healthy again as evidence. This would be evidence that backs up the claim that "Tamiflu cures flu" in a pseudo-scientific fashion, and would be clearly flawed - would I have recovered without Tamiflu?

Against that, I might claim that my camera "sees" infrared light; the test would be to get a high intensity IR source, and if it didn't show up in a photo taken by my camera, I'd know my claim was wrong. If it shows up, then that reinforces the claim, and it's up to me to try another way of disproving it; as I struggle to find a way to disprove it that works, it becomes less and less likely that my claim is false.

The challenge, then, for people presenting "scientific" claims, is to show that they are backed by evidence, and that the evidence has been gathered by trying to prove the claim wrong. In an ideal world, you describe why you gathered the evidence you did, and how I could gather it myself, so that I can check your claim. I can then be confident in your claim, knowing that if I start to disagree, I can check it.