"A hopeless quest" - Michael Blastland and Andrew Trotter discuss accurate reporting of crime statistics
I recently went to an event entitled “Data and news sourcing” jointly organised by the Media Standards Trust and the BBC College of Journalism, and hosted by the Royal Statistical Society. So far I’ve written a couple of blog posts based on a panel session that featured Paul Lewis talking about crowd-sourcing journalism alongside Paul Bradshaw and Turi Munthe, which also saw me interjecting to talk about how to measure the success of the Guardian’s MP’s expenses app. The other panel I attended looked at the reporting of crime statistics, and featured Michael Blastland and Andrew Trotter amongst others.
Michael Blastland
Michael said he considered himself “a semi-detatched journalist”, as he spends more time on statistics now than reporting. He asked for a show of hands in the room of anybody who had ever done anything illegal. With everybody’s hand held aloft, he pointed out that no crime survey ever suggests that there is 100% criminality in society, and so even on that basis we ought to understand that the numbers are flawed. Whatever numbers we have, he said, can only ever be a “slice” of the real numbers about crime, and we have no way of truly knowing how accurate that slice is.
Not only can we never have entirely accurate numbers, but the media puts a spin on them. Blastland cited a spate of tabloid headlines suggesting that one in four teenage boys was a persistent violent offender, and then showed us that this figure was derived from counting up those who had said yes to a question in a survey which defined violent offending as “scratching, hitting, kicking, throwing things” and told the kids to say yes if they had ever done that to family members.
As part of his talk, Michael put quite a few graphs and representations on screen, which for the first time in a long while made me wish I’d tried a bit harder at maths when I was at school. He said that society was “twitchy” about crime figures, always looking for trends, and the media made stories out of what they saw to be trends, even if the movements in the figures were well within the margins of error or “confidence levels” in any given set of statistics.
Another thing he stressed was that clustering naturally occurs in all datasets. A while ago the BBC had a story that four people had been stabbed in one day in London. Michael said that with 170 stabbings a year in the capital, the balance of probabilities was that every three years, four of them would take place on a single day.
Andrew Trotter
Andrew Trotter is Chief Constable of the British Transport Police, and has a role in media liaison for the police as a profession. He opened by saying that he thought getting accurate and rational reporting around crime figures was “a hopeless quest”.
Politicians, journalists, and sometimes the police themselves abuse crime figures he said. Politicians use any movement in the numbers to prove that their policy works, or someone else’s doesn’t. He argued that we regularly see media headlines where journalists use very small movements in numbers to prove or disprove an editorial line that their paper has already taken story.
And police officers are guilty of that too, Trotter said, for example claiming that some specific operation drove numbers down, without accounting for longer term factors. He recognised that there is no scientific rigour in isolating the factors that might be causing changes in crime figures.
Andrew Trotter had some good examples of where failing to look properly into the data led to the wrong conclusions being drawn. He cited one train station appearing to be head and shoulders above other South London stations in terms of the crime committed there, even though it was in a leafy and relatively crime-free suburban part of town. It turned out that this was a statistical anomaly caused by one railway worker obsessively reporting even the slightest scratch to the station as “criminal damage”.
Trotter thought that the publication of more data and the recent release of crime maps was a good thing. He claimed it has provoked a more intelligent debate about crime, and there seemed to be a new reticence from politicians to jump straight into things without a bit more fact-checking. He argued that new media means people are much more sceptical, and that they can more easily reference source documents and refute spurious arguments based on a bad interpretation of the data. All of us, he said, meaning the police, politicians, and journalists, can no longer get away with some of the stuff they used to, because they are now being policed themselves by a bunch of intelligent observers across the internet.
Next...
Also on the panel discussing crime statistics was BBC Home Affairs correspondent Dominic Casciani, and Telegraph reporter Conrad Quilty-Harper, I’ll be blogging my notes from their talks when I get the chance.