Australian government makes datasets publicly available
March 7, 2012 7:46 PM Subscribe
Data.gov.au is a site giving public access to datasets from the Australian federal, state and territory governments. It was created in response to the Declaration of Open Government, which aims to get more citizen collaboration in policy and service delivery design. People are encouraged to use these datasets to produce apps or conduct research. So far the little-publicised site has resulted in apps such as Dunny Directory, Convict Records of Australia and Transhub, a public transport planner for the nation’s capital. If you’re interested in more online government participation in Australia, Craig Thomler is tracking developments on his eGov AU blog.
That eGov blog is saying things that are very dear to my heart. I need to grab a lot of government data as part of my job - not just Australian government data, but often US data as well. And the Americans do it so much better! Putting the data out there, online, in raw format is just the default for most US agencies I deal with - I think it's a result of the "my taxes paid for it, so you'd better let me see it" attitude.
In contrast, obtaining data from Australian government agencies is an exercise in anguish and expense. You email to find out what data is available, you email for a quote for "cost recovery" (a fee that often has little relevance to how much time would actually be spent getting you the data), you often have to sign a data agreement (in some cases, wait for one to be specially drawn up, then you hand it over to your lawyers). Then you receive a bunch of, in my case, ESRI shapefiles with out-of-date metadata and incorrectly defined spatial projections...
Data.gov.au is heading in the right direction, though. There's some useful stuff on there - there are also gaping, obvious holes (meteorology is basically limited to current weather warnings - no historic data, no climate data...)
What we need is not so much a central repository, we need a change in both legislation and public service culture, so that publicly funded data is made available by default, automatically.
posted by Jimbob at 8:27 PM on March 7, 2012 [1 favorite]
In contrast, obtaining data from Australian government agencies is an exercise in anguish and expense. You email to find out what data is available, you email for a quote for "cost recovery" (a fee that often has little relevance to how much time would actually be spent getting you the data), you often have to sign a data agreement (in some cases, wait for one to be specially drawn up, then you hand it over to your lawyers). Then you receive a bunch of, in my case, ESRI shapefiles with out-of-date metadata and incorrectly defined spatial projections...
Data.gov.au is heading in the right direction, though. There's some useful stuff on there - there are also gaping, obvious holes (meteorology is basically limited to current weather warnings - no historic data, no climate data...)
What we need is not so much a central repository, we need a change in both legislation and public service culture, so that publicly funded data is made available by default, automatically.
posted by Jimbob at 8:27 PM on March 7, 2012 [1 favorite]
Open data is awesome!
Incidentally, if you've got a government you want to convince of that: One benefit of making data sets open to the public is that it also opens them to other departments and agencies. It's a way of breaking down silos, making it so the public health department can see and use the data collected by the transport department immediately, instead of after weeks/months and through layers of bureaucracy (if at all).
posted by parudox at 8:42 PM on March 7, 2012
Incidentally, if you've got a government you want to convince of that: One benefit of making data sets open to the public is that it also opens them to other departments and agencies. It's a way of breaking down silos, making it so the public health department can see and use the data collected by the transport department immediately, instead of after weeks/months and through layers of bureaucracy (if at all).
posted by parudox at 8:42 PM on March 7, 2012
Knowledge only means complicity in guilt; ignorance has a certain dignity.
posted by pompomtom at 8:49 PM on March 7, 2012
posted by pompomtom at 8:49 PM on March 7, 2012
Oh wow, thanks for that Convict Records of Australia link - took all of two seconds to find my convict ancestor. Naughty Henry, fell in love with a prostitute and stole presents for her from his workplace; having been to London recently, we are super grateful he did.
posted by Wantok at 8:56 PM on March 7, 2012
posted by Wantok at 8:56 PM on March 7, 2012
Jimbob: "What we need is not so much a central repository, we need a change in both legislation and public service culture …"
Agreed. On top of that, though, there needs to be some institutional understanding of data curation and access.
It's been my misfortune lately to try and work with some data from OneSchool, which is itself a WTF. Some of it is available as raw dumps (which at least has the advantage of being searchable, if unwieldy), most of it is available as Excel files only through a limited web-based query interface, and a frustratingly important amount is available only as generated PDFs which are useless to import from. Most of it is full of holes anyway. And that's with full teacher / institution access.
I'm sure in that case the idea is to make it difficult for people to leak data that can be analysed for use as ammunition - but when a teacher is asked something as simple as "project this cohort's performance for the next semester" and can't, because relevant data can't be gathered coherently…
Didn't historical data from the BoM used to be easily available off the website? I'm sure I used 1~2 year historic daily summary data a few times during my undergrad, but I've got a friend currently doing some research that could benefit from daily or weekly data (even summaries) who tells me that's no longer the case.
posted by Pinback at 9:05 PM on March 7, 2012
Agreed. On top of that, though, there needs to be some institutional understanding of data curation and access.
It's been my misfortune lately to try and work with some data from OneSchool, which is itself a WTF. Some of it is available as raw dumps (which at least has the advantage of being searchable, if unwieldy), most of it is available as Excel files only through a limited web-based query interface, and a frustratingly important amount is available only as generated PDFs which are useless to import from. Most of it is full of holes anyway. And that's with full teacher / institution access.
I'm sure in that case the idea is to make it difficult for people to leak data that can be analysed for use as ammunition - but when a teacher is asked something as simple as "project this cohort's performance for the next semester" and can't, because relevant data can't be gathered coherently…
Didn't historical data from the BoM used to be easily available off the website? I'm sure I used 1~2 year historic daily summary data a few times during my undergrad, but I've got a friend currently doing some research that could benefit from daily or weekly data (even summaries) who tells me that's no longer the case.
posted by Pinback at 9:05 PM on March 7, 2012
What we need is not so much a central repository, we need a change in both legislation and public service culture, so that publicly funded data is made available by default, automatically.
Have you seen the Principles on Open Public Sector Information put out by the Office of the Australian Information Commissioner?
posted by His thoughts were red thoughts at 9:34 PM on March 7, 2012
Have you seen the Principles on Open Public Sector Information put out by the Office of the Australian Information Commissioner?
posted by His thoughts were red thoughts at 9:34 PM on March 7, 2012
Pinback, station-based data is only available for a few weeks in the past - for longer term archives you have to email them and buy it. However, the price is per-request not per-amount of data, so if you're buying met data, you may as well ask for everything.
Another option is the Australian Water Availability Project, which has gridded/interpolated data which is freely available online, but the interface it provides is more geared towards looking at the whole country for a single day, rather than a time-series of days for a single location.
posted by Jimbob at 9:46 PM on March 7, 2012 [1 favorite]
Another option is the Australian Water Availability Project, which has gridded/interpolated data which is freely available online, but the interface it provides is more geared towards looking at the whole country for a single day, rather than a time-series of days for a single location.
posted by Jimbob at 9:46 PM on March 7, 2012 [1 favorite]
Oh wow pretty much my entire family tree is convicts. It's convicts all the way down. Now I know why it is that I can't help but shoplift stuff all the time.
posted by tumid dahlia at 9:58 PM on March 7, 2012 [1 favorite]
posted by tumid dahlia at 9:58 PM on March 7, 2012 [1 favorite]
Thanks for the link to the sacking, Pinback, I wasn't aware of that. What a short-sighted move. Seems relevant to include this then, which I didn't because I couldn't manage it without editorialising :) The paranoia that will 'shut' government.
posted by harriet vane at 10:03 PM on March 7, 2012
posted by harriet vane at 10:03 PM on March 7, 2012
Have I offended you in a some way tumid dahlia or are you just a dick?
posted by Wantok at 11:45 PM on March 7, 2012
posted by Wantok at 11:45 PM on March 7, 2012
And the Americans do it so much better! Putting the data out there, online, in raw format is just the default for most US agencies I deal with - I think it's a result of the "my taxes paid for it, so you'd better let me see it" attitude.
I do the same for Canadian and US data, and I couldn't agree more with you. I had to sign an 8 page legal document, mail it to the government, then wait for DVDs to get mailed to me, for the exact same Canadian Census product that I downloaded in .csv format from the US Census Bureau this afternoon. Although I think Statscan is trending positive.
Data availability is one of the few non-military functions where the US government shines.
posted by Homeboy Trouble at 11:56 PM on March 7, 2012
I do the same for Canadian and US data, and I couldn't agree more with you. I had to sign an 8 page legal document, mail it to the government, then wait for DVDs to get mailed to me, for the exact same Canadian Census product that I downloaded in .csv format from the US Census Bureau this afternoon. Although I think Statscan is trending positive.
Data availability is one of the few non-military functions where the US government shines.
posted by Homeboy Trouble at 11:56 PM on March 7, 2012
a public transport planner for the nation’s capital
Bus.
Yes.
Yes.
posted by obiwanwasabi at 12:04 AM on March 8, 2012 [6 favorites]
Would you like to travel by bus, bus, or bus?Bus.
Would you like to catch a bus today?Yes.
Would you like to get to your destination today?Yes.
NO ROUTES FOUNDposted by obiwanwasabi at 12:04 AM on March 8, 2012 [6 favorites]
Have I offended you in a some way tumid dahlia or are you just a dick?
Sounds like he's offended you, Pinback. Though reading his comment I have no idea why you'd feel like that.
posted by Kerasia at 3:36 AM on March 8, 2012
Sounds like he's offended you, Pinback. Though reading his comment I have no idea why you'd feel like that.
posted by Kerasia at 3:36 AM on March 8, 2012
I got the impression that PinbackWantok got offended by tumid dahlia, because Wantok denigrated London and thought they were responding to that playground taunt.
Wantok suggested that tumid dahlia was "just a dick", which is a kind of sledging and reflects poorly on the accuser.
Basically, what I'm trying to say is, as an Australian who has spent a few years living in London. Wantok is just a dick.
posted by panaceanot at 4:13 AM on March 8, 2012
Wantok suggested that tumid dahlia was "just a dick", which is a kind of sledging and reflects poorly on the accuser.
Basically, what I'm trying to say is, as an Australian who has spent a few years living in London. Wantok is just a dick.
posted by panaceanot at 4:13 AM on March 8, 2012
Another thing for the incoming Abbott government to kill for reasons of culture-war totemism (i.e., giving information away is socialist and evil, selling it on the free market is right and good), right after they scrap the carbon trading scheme, make mining tax-free and privatise the Great Barrier Reef.
posted by acb at 4:46 AM on March 8, 2012
posted by acb at 4:46 AM on March 8, 2012
The thing is, acb, experience bears out that it takes a capitalist, tax-hating, guv'mint hating place like the US to produce open data availability, since the citizens seem to demand to see the "results" of where their tax dollars are spent. Examples are plentiful. Look at how all publicly-funded health research in the US has to be made available in open-access journals. Look, in contrast, with the clusterfuck that is the Ordnance Survey in the UK.
posted by Jimbob at 5:20 AM on March 8, 2012
posted by Jimbob at 5:20 AM on March 8, 2012
Seriously now, which one of you bastards called this bastard a bastard?
Enquiring minds need to know.
posted by Hello, I'm David McGahan at 5:29 AM on March 8, 2012
Enquiring minds need to know.
posted by Hello, I'm David McGahan at 5:29 AM on March 8, 2012
Basically, what I'm trying to say is, as an Australian who has spent a few years living in London. Wantok is just a dick.
posted by panaceanot at 4:13 AM on March 8 [+] [!]
Sir! I am offended! Many of my good friends are dicks! Also they're lazy layabouts, etc...
But Still!!
posted by Hello, I'm David McGahan at 6:02 AM on March 8, 2012
posted by panaceanot at 4:13 AM on March 8 [+] [!]
Sir! I am offended! Many of my good friends are dicks! Also they're lazy layabouts, etc...
But Still!!
posted by Hello, I'm David McGahan at 6:02 AM on March 8, 2012
« Older More Short Stories | Tetsuo's final scene from Akira + Mardi Gras Newer »
This thread has been archived and is closed to new comments
posted by Pinback at 8:05 PM on March 7, 2012 [3 favorites]