ISF interview with Professor Sir David Spiegelhalter

Professor Sir David Spiegelhalter is Chair of the Winton Centre for Risk and Evidence Communication at Cambridge University. Crucially, he is also currently the Joint Chair of the Royal Statistical Society COVID-19 task force. He tells Saira Shah about the mathematical and real world underpinnings which inform his work on the pandemic and other statistical risks. He also illustrates why we need to use both our instinctive gut reactions in tandem with slow, methodical thinking to make sense of modern risks.

Interview with David Spiegelhalter and Saira Shah

Professor Sir David Spiegelhalter has been called probably the greatest living statistical communicator. He's devoted much of his career to helping the public understand risk. He's Chair of the Winton Center for Risk and Evidence Communication at Cambridge University. His rare gift for explaining how to use data, to understand the probabilities of various risks is evidenced by his brilliantly readable book, The Art of Statistics.

So David, we're so lucky to have you with us today. And, I know that you're currently Joint Chair of the Royal Statistical Society, COVID-19 Task Force, and you're very, very in demand. So thank you doubly for your kindness in joining us today.

Can I ask you, we've seen a lot of psychological research that shows we tend to make judgments about risk based on emotions rather than facts.

1. Why do we need statistics?

Oh, that's a good question. I'm a statistician, so I started off as a mathematician. But most of my work now is collaborations with psychologists and I find this such an exciting way to work. Essentially it's not a choice between an emotional gut reaction to risk and a mathematical probabilistic approach to risk. We all know about Daniel Kahneman's distinction between thinking fast and slow, whether you call it type one/type two thing, whatever you want to call it. There are these two broad ways of approaching a problem.

One which is very much to go with your first feelings, your gut reactions. And the other is just to try to slow down and think through things methodically. And neither is right. Neither is enough. I think humans are a wonderful combination between the two.

It's so exciting to have those two perspectives to take on a problem. People have said that you've got risks as analysis and you've got risk as feeling, and they're both terribly important. This interaction between those - that's where I work quite a lot - which is so riveting, particularly at this time of COVID, but actually in all our circumstances in everything we do every day.

2. Can you give me some examples of the common mistakes that the public and journalists tend to make when faced with figures about risks?

First of all I don't like to say that people make mistakes. I'm never going to say someone makes mistakes about their feelings, because people have fears and anxieties.

I've got mine. I suffer from every bizarre personal bias or whatever. I've got it. We've all got it. So it's not like I'm saying people are wrong in terms of their feelings. There are some things people are just wrong at when they do try to reason, when they think they're being reasonable about numbers, whatever, and yet they get hopelessly wrong.

The two famous ones that people have identified. The first one is really about relative risks and absolute risks. We always share stories about something doubling the risk and that sounds frightening.

Well, is it? Or not? We just don't know. Double very little… is still very little. We have to know double what? And once we say well, if that thing is doubling, it's actually extremely rare: one in a million chance of this happening in your lifetime or over a year. Well, who cares? So what, it's just not important.

3. You had a great example of that in your book about bacon sandwiches.

The bacon sandwich is a standard one: a daily bacon sandwich increasing your risk of bowel cancer by 20%. Let’s assume that’s true. Well, it increases it from what? About six percentage points over your lifetime. And so a 20% increase over that takes us up to about seven percentage points. So that means that a hundred people are going to have to eat a bacon sandwich every day of their lives to get one extra case of bowel cancer. That puts it in perspective.

Saira Shah: I’m going back on the bacon now (laughs).

David Spiegelhalter: Oh, yeah, so pass the brown sauce is the natural thing. You know, I do like bacon and I must say from my purely emotional response, when I looked at that data, I did cut down on the amount of bacon I ate. I wasn't going to eat it every day, but I still will eat it.

It's not like it's now a forbidden thing. It's just one of those cream cake treats. The other one of course is in diagnostic testing. We're seeing that a lot in COVID: the idea of false positives. Those ideas can be deeply unintuitive, such as the recent claim that most of the positive tests for COVID are false positives, even though the test is quite accurate. Actually, that can be true under certain assumptions.

4. Can you talk me through false positives?

The claim is that most of the positive tests are false positives. That's based on the claim, which was made by the minister that the COVID test has a false positive rate of 1%.

That means that out of a hundred people who have NOT got a virus… one will test positive. Okay. Let's assume that's true for the moment. It isn't, but let's assume it's true.

Now let's say that you're in the situation that we were in this summer where any one in a thousand people had the virus. And you test a thousand people on average.

You'll probably get the one who's got it, but you've got a thousand or 999 others who haven't got it. And if 1% of them are going to test positive as a mistake, that's 10 of those who will test positive. So you've got 11 positives and only one of them is actually a true positive.

Saira Shah: Got you, I get you.

David Spiegelhalter: That's the situation that could have occurred during the summer when the virus was very rare if you were using this test. But there's two things that are wrong with that.

So it is theoretically possible that that happens. That's not the situation at the moment. First of all, the people getting tests have nearly all got some symptoms. For people who have got symptoms, the proportion who have got the virus isn't one in a thousand, it's about 5% or something like that.

So actually that will increase hugely the number of true positives. But the other thing is that the test’s false positive rate was way below 1%. It's more like 0.05%, one in 2000, not one in a hundred for a PCR test.

Two reasons why at the moment most of the people who get a positive test have got the virus -- doesn't mean they're infectious, but they have got the virus. So what that shows is a little reasoning can also be a bad thing because the mathematics of that false positive was correct. It's just the assumptions were completely wrong.

Basically, anything to do with diagnostic testing is really tricky. It's the one of the worst ones for intuition. And this is where you can't think fast. I have to get out a piece of paper and start writing out the numbers.

My intuition is hopeless on this as much as anybody else's. So, the probability is tricky. I've said it so many times probably, but people ask me, why do people find probability and statistics so unintuitive and difficult.

Nearly 50 years I've been working in this area and I finally concluded that it's because probability in statistics really are unintuitive and difficult.

5. I'm a daughter of in part of Tunbridge Wells among other places. And now I realized that Tunbridge Wells is famous for a certain figure, who is an enormous name in probability. Can you tell me about that?

The famous non-conformist clergyman, Thomas Bayes, who lived there. Apparently he was an appalling clergyman. His sermons were outrageously dull, apparently. He was 18th century and died in 1761, a fellow of the Royal Society. He wrote an absolutely impenetrable, appallingly-written paper. Although it did have the very vivid thought experiment behind it, which introduced this idea of using probability theory to learn from experience. That probabilities don't exist out there in the world.

Basically they are a way of dealing with our own personal uncertainty about facts and then as we receive new information, then our probabilities are revised, according to what is now called the Bayes Theorem after the non-conformist boring vicar from Tunbridge Wells, who's a fantastic figure in the history. He lapsed into obscurity, because, although his paper was published, no one took much notice of it and Pierre-Simon Laplace discovered the same results 10-15 years later and wrote it absolutely beautifully, with wonderful clarity and so he got the credit essentially.

6. Does that help us around your problem of intuition versus statistics?

Yes, it does. The way Bayesian statistics tells us the formally correct mathematically correct way to learn from experience. And so it's a wonderful result, and I view it as an ideal. However, it is unachievable ideal because it kind of assumes that you or your understanding is correct from the beginning.

It assumes you have good knowledge about how the world works and essentially that your model is correct. In other words, the structure in which you are putting your probabilities is appropriate. But of course, that’s wrong.

7. Do you have a really simple anecdote or example to explain that kind of thinking, because it’s difficult to understand.

Oh, well of course, what I've just talked about about the COVID test is a perfect example. That we cannot interpret a COVID test when it's positive without knowing who it's being tested on. In other words, how common was COVID in the population in the first place.

So in other words, if I tell you my COVID test was positive, the chance that I really do have the virus – to assess that you need extra information. You have to know what was the chance I'd had the virus before I took the test, and then I get the positive test. And then using Bayes’ Theorem, that probability is updated to the new probability…. which is actually called a posterior probability after the test results come in. So for example, with a random person being tested, there was a one in a thousand chance they had the virus, but actually after the positive test came in, it had gone up to about one in 10.

8. So you have worked out how to take life into account in a mathematical way.

There is a mathematically correct way of updating our uncertainties on the basis of new knowledge. However…… It sounds great doesn't it? Oh, well that's all we have to do. No, because as humans we do have some understanding of how the world works. Because if there are possibilities that we never even thought of, then we’re messed up.

We'll get more and more confident, but actually we are being completely deluded because our assumptions were wrong. So this is, again, what I wanted to introduce, the idea about the COVID thing. Being mathematically correct is not enough if our basic assumptions, our understanding of the world and how it works, is just wrong.

And that is something that, within this COVID debate, obviously clearly is crucially important.

9. What are some of the real life costs of making basic mistakes in probability? And maybe we can put our assumptions aside for a moment and assume you've got more assumptions, right. But in, actually making the statistical mistakes.

Well, I would say getting your assumptions wrong is making a statistical mistake. And usually with software, things like that, the actual calculations will be appropriate. I mean, people can make actual errors in the calculations, but they're generally correct. So it's all to do with the fact that our assumptions are wrong.

A classic one was the financial collapse from 2008. And I know it's not holding you to it, but a lot of that was due to the fact that people have these financial models and they believe them. The models work correctly. They said all the things that we're supposed to do and they'd worked in the past. It's just that the assumptions basically stopped being appropriate. And they were just saying crazy things that caused enormous financial damage because people were making decisions on the basis of inappropriate models.

So again, I come back to a wonderful phrase which I'm sure most people know of. The famous statistician, George Box, said: 'All models are wrong, but some are useful.' And it's a rather cliched phrase, it’s got its own Wikipedia page, but you just have to say it again and again and again that everything we claim is on the basis of our understanding of the world. And if that understanding is wrong, then our claims are inappropriate. And, any mathematical representation of the world is always wrong. It's never absolutely correct, but it can be useful. And we're seeing this all the time with these epidemic models, which are all wrong. They’re all gross simplifications and people say they are like the map rather than the actual territory. But they can be useful if treated very, very, very cautiously.

10. Now I'm going to ask you a really tough philosophical question: Given we are in this world where we're being swamped with more and more data at every moment - and our assumptions are bound to be wrong - how can we navigate our way through this increasingly complex world?

It's very difficult because we're all playing different roles in this. You've got the actual analysts, a bit like me. I'm looking at the data, finding it very difficult to negotiate through which ones do you believe, which is the best quality evidence, do the numbers mean what they say?

What is a COVID death anyway? We are trying to put those numbers in context. Is this number big or small because you can't judge by numbers without putting them in some sort of context. And then you've got the decision makers, the politicians and people like that. They've got to try to interpret all this stuff and make sense of it and that's a real challenge.

And then you've got the public, which I feel very sorry for them being bombarded with rival claims and analysis of the newspapers full up with graphs and statistics, stuff like this. And all drawing often very … different conclusions.

So, it is a real challenge and there's no simple answer to this at all. I mean, I can say what I do because I spend my time critiquing evidence, which is very much to look at the trustworthiness of the source. Essentially, what are the motivations? Even before looking at data, I ask, why am I hearing this?

What are the motivations of the person telling you the story? Are they trying to convince me of what they think or are they generally trying to inform me and enable me to make a better decision? And that's the first thing I ask: what are the motivations of the person who is telling me, or my source of the information or evidence.

And if I know that that's coming from a source that has already made up its mind and is trying to persuade people of something, then I really struggle to take much notice of what they're saying or what this data is saying. I'll look at it, but I'll always be thinking, ‘Well, what am I not being told?’

What have they cherry picked? So, that's actually really quite challenging to do that, to spot when someone's cherry picking, because you have to know what am I not seeing? And there's one of the most difficult things. It's easy to critique what you've been told, it's being able to critique what you're not being told. It’s one of those difficult challenges there is in this area.