The reporting of Internet polls during elections

Martin Belam  by Martin Belam, 10 June 2004

Well, the voting is over - so I can come out of hiding. There is something as yet unresolved about identifiable BBC staff writing about an election camapign.

Within the BBC, for the duration of an election campaign we go into "special conditions of operation" mode. This is less exciting than it sounds - but it means that we are extra careful about the user generated content we publish and the opinions expressed on bbc.co.uk pages. At no time is the BBC's committment to impartiality more vital than during an election campaign. During it we face some tough choices. Should bbc.co.uk/news link to the BNP website from its Vote2004 site? Can we allow users on the messageboards to discuss the election unfettered?

I made my own less tough decision that I didn't think it was appropriate to write about the politics of the elections, apart from linking to the election promotion sites like aboutmyvote and londonelects, during the course of the campaign.

However, with voting closed I can now talk about something I've been wanting to - which is the use of internet polls in producing the news agenda, and my worries about the way it is reported. The eve of poll front page splash from the Evening Standard was that the London Mayoral race was neck and neck, based on a poll conducted by YouGov.

Now currently just over half of the households in the UK have internet access. The YouGov poll is conducted by questioning a self-selecting group of these. We can debate the merits of sampling in political polls another day, but the important thing is that if you are sampling you need some process with which make the results representative of the population as a whole. I can't see any way that you can reasonably extrapolate the results of an online poll to cover the offline population.

This isn't to knock YouGov in any way - but just to question whether other media outlets are being as cautious with what I would consider to be "dirty data". The BBC is very clear about where and when internet voting figures should be used in programmes. The online producer's guidelines state that:

Interactivity of every sort is part of the central appeal of any online site. However, inviting people to express their views on any issue by its very nature involves a self-selecting sample and is representative of nothing more than those people who have chosen to respond. On BBC sites, especially News Online and programme sites which may relate to political or public policy issues, care has to be taken that online expressions of opinion are not translated into anything that could be construed either as an accurate representation of public opinion as a whole or as the BBC's opinion. So any summary of online voting or expression of opinion should:

- not be called a poll

- not be expressed solely in percentage terms. The results should be expressed primarily in terms of how many hits the yes or agree button has received and how many the no or disagree button. An exception to this may be made for a vote which is not concerned with a political or public policy matter. If only percentages are to be used, the approval of the Chief Political Adviser must be sought in advance

- any online summary of online voting or expression of opinion about political or public policy issues must include a disclaimer to the effect that "This can't be a representative sample and the figures don't necessarily represent public opinion".

To their credit YouGov include something similar on their open polls

Note to media: results from surveys on this page should NOT be taken as representative of UK public opinion, and should never be published as such.

I'm just not convinced headlines like this convey the frailty of using a closed entry online sample of 1,517 Londoners.

A picture of the Evening Standard front page from Wednesday 9th June 2004 proclaiming that the Mayor Race is Neck and Neck, based on a sample of 1,517 people

3 Comments

Martin - another point is that this self-selecting group is already only representative of a certain sample of the UK population: UK internet users. If you check out the demographic, salary bands and psychographic profiles publicly available for UK internet users, you'll still see some quite heavy skewing towards younger, wealthier types. So it's a case of self-selection from a not necessarily representative pool: a bit like Whiskas running a poll delivering results saying "most people prefer cats"!

Out of interest, the special conditions of operation don't stop until Monday morning.

Once upon a time I actually wanted to be a psephologist (no really). So one of my heroes has always been Peter Kellner which is why I have a soft spot for Yougov. Apart from the fact that have been accurate (they had the mayor race pretty much spot on - in the Sky News poll they conducted on the day).

Internet polls obviously have a bias but then so do phone polls and they are all carefully sampled and I think Peter here does make the point that you can extrapolate from internet polls an effective representation of the offline population quite well. But then I've got a soft spot for him haven't I.

Keep up to date on my new blog