What open Government data gives us with one hand, closed state data takes away with another

 by Martin Belam, 24 November 2010

The oddest thing to me about the release of Government department spending data last week is the way it 'feels'. It feels like it is being published not as 'a good thing to do' in itself, but as a corrective measure. As if, instead of treating the civil service as the instrument of government, the Coalition is treating them like 'the enemy within', who cannot be trusted with money unless the threat of public exposure hangs over them.

In an interesting blog post earlier this week, Simon Briscoe pointed out some of the obvious limitations with the data as it is currently being published. There is little context, some obvious duplication, and no way of knowing whether the sums involved cover projects or services of short, medium or long-term duration. And until we have month after month after month of comparable data, we cannot draw any firm trends from the numbers.

There was another issue I saw, which goes to the heart of building a linked data ecosphere on the web.

Using the Guardian's data explorer tool, you can get a comprehensive list of suppliers. Wouldn't it be wonderful if you could instantly cross-reference that with the records at Companies House?

I'd love to be able to get an instant snapshot of how many of these companies are large, medium or small enterprises. Over time you could use that to measure whether the intention to open up Government service tendering to wider competition was on track or not.

It would also be interesting to compare the sums being paid to a company's declared accounts. What sort of proportion of their turnover derives from state spending? That would help identify companies who were almost solely dependent on state subsidy for their existence.

So how would you do that with linked data?

Well, ideally you'd want the spending data dump to feature a company's registered number. Or at least a unique supplier number that could be mapped to a Companies House record.

But, here comes the barrier.

All of this data is available via WebCHeck online from Companies House, but it is behind a paywall. It is £1 to get a general record, and an additional £1 to get different types of detailed records like accounts for each company. So what open Government data gives us with one hand, closed state data takes away with another. You can do all that cross-referencing, but it is a job for some researchers with access to a budget, rather than a dataset that could be compiled pretty much automatically.

But let us not be too negative.

Whatever the ideology behind the release of the spending data, and for all the faults that data so far has, this Government does seem to be committed to the open release of data in a way that no other has been before, and they should be applauded for it.


I've never trusted Government and it's sources, you would never find properly and honestly written information there, after all they, I mean Government, are interested in spreading false info...

I agree that the GBP 1 charge for a copy of the accounts from Companies House may to some degree reduce the public's access to this data. However, in my experience the "small company scheme" under which the data included in the accounts can be significantly reduced, is more of a danger to public access to information. First of all, a company is considered small up to a turnover of GBP 6.5 million - which is in fact not that small - and more importantly, "single purpose companies" are often used by larger companies to reduce the turnover to less than this threshold and thus hide details from competitors and the general public.

Keep up to date on my new blog