it.gen.nz

I’ve always been interested in this. We have already paid our government to collect data – why should it lock the data away from us, or charge us for access? Just because you have the power to do something, doesn’t make it right, after all. And, to be fair, the New Zealand government has been publishing a lot of data for a long time.

I first got involved in this space back in about 1994 when my then-boss in the Ministry of Commerce encouraged me to go out and see if I could develop a government web server. (Back in those days, of course, the Web was brand new and unknown to most people.)

The technology wasn’t a problem, or not my problem anyway. I hooked up with Nathan Torkington at Victoria University, who had also been pushing the idea of a government web server, and he persuaded the Uni to provide space on a server. To this day, I’m not sure if the Uni knows it did that, but thanks anyway.

I rapidly found that the biggest problem was sourcing content for the server. I developed a “pitch” to government departments saying: give me your documents and your data for publication – I’ll do the rest. There were all sorts of objections. Some financial, some that they weren’t happy with quality, and some that they couldn’t manage who was getting copies. Incidentally, the Official Information Act answers all of these. But many, even most, government folk I spoke to either saw the opportunity, or believed in the values of open information enough to help. Really important content that I couldn’t get any other way, I simply wrote.

The State Services Commission came to the party in 1997 with policy, since revised and summarized as the Policy Framework for Government Held Information. This enshrines the principles that government departments should be making more and more information available, and should not be charging people more than the cost of publication. And in a Web environment, the marginal cost of publication is zero.

Even the title of the SSC policy with its reference to “Government Held” information makes clear where its coming from – government collects information on our behalf, it doesn’t “own” it.

This is all background to something I ran across on the Net the other day. An NGO called Public.Resource.Org got together 30 e-government advocates, all from the private sector, academe or NGOs, and probably all from the US. They came up with a set of principles for open government data. This was written last year, well into the age of reuse and Web 2.0. (It’s worth noting that Public.Resource.Org is led by the highly-credible Carl Malamud who has been doing great things to open up US government data.)

The principles they came up with are:

Government data shall be considered open if it is made public in a way that complies with the principles below:

1. Complete
All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations.

2. Primary
Data is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms.

3. Timely
Data is made available as quickly as necessary to preserve the value of the data.

4. Accessible
Data is available to the widest range of users for the widest range of purposes.

5. Machine processable
Data is reasonably structured to allow automated processing.

6. Non-discriminatory
Data is available to anyone, with no requirement of registration.

7. Non-proprietary
Data is available in a format over which no entity has exclusive control.

8. License-free
Data is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed.

Compliance must be reviewable.

These principles seem reasonably obvious from a technology and policy perspective. Let’s just look at where New Zealand policy is against them:

1. Complete – check.
2. Primary – no. There’s no mention of this in NZ policy.
3. Timely – no, although you could run an argument for it based on the policy.
4. Accessible – there is a wider policy about making government information online in accessible formats. See the Government Web Standards.
5. Machine processable – not explicitly, although the Web Standards imply that data should be published in simple formats that can easily be scraped. PDFs are definitely frowned upon.
6. Non-discriminatory – not mentioned in NZ policy, but is strongly implied by the Official Information Act. Any attempt to discriminate would be open to challenge.
7. Non-proprietary – not mentioned in NZ policy, but the e-GIF does go some way towards this. It’s not always honoured by departments publishing information, though.
8. Licence-free – no. In New Zealand, information published by the government is subject to Crown Copyright – even though the policy notes that the government doesn’t own it. I’m not clear how this is possible. In practice, the Crown Copyright statements generally liberal use of the information provided.

So, in some areas (particularly copyright) the New Zealand government policy falls short of the American principles.

But another question to ask is: can the SSC policy, which has been around for over a decade, inform the PublicResource.Org principles?

The two that strike me on re-reading the SSC policy, are the distinction between ownership and stewardship and the statement of privacy principles. Ownership and stewardship are an attempt to get round the whole ownership of data problem. Stewardship means looking after something on behalf of its owner, which, in the case of government-held data is generally the public. That’s how we end up with Crown Copyright statements on everything.

Privacy principles are sufficiently important in my view to merit explicit statement in the SSC policy. They come directly from the Privacy Act. Having generic privacy principles in law is a significant difference between the New Zealand and US privacy environment, or between the US and pretty much anywhere for that matter.

To wrap up – it’s good to see the level of agreement between these. However, I do worry about some aspects of NZ government policy and whether they can permit the kind of re-use Web 2.0 offers. In particular I am concerned about the varying Crown Copyright terms that departments release under, and the variety of proprietary and non-proprietary formats, especially non-processable ones, that data appears in.