One year ago, the U.S. government launched Data.gov, a central plank in its Open Government initiative to make it easier for the public to find and use official datasets. The site has grown from an initial 47 databases to more than 272,000, and attracted nearly 100 million hits. It inspired eight American cities – including San Francisco and New York City – eight states, and six other nations to launch similar sites of their own. By most metrics, the project has been a success.
But government transparency advocates say the site has plenty of room for improvement. Gabriela Schneider ...
Whither Transparency
The New York Times this morning took the Administration to task because White House officials and lobbyists are meeting at... View Article
Continue readingNational Data Catalog API
The National Data Catalog went live last week. Now we would like to share a little bit about our API and how it fits into our platform.
The National Data Catalog (NDC) is an open source catalog for government data sets and APIs. Our goal is to have it encompass all data released by or about governments in the United States. This includes federal, state, and local jurisdictions. The NDC will harness the community of users interested in open government data.
Web developers can take a look at our API documentation.
Continue readingIllegal immigrant stats available, but yet to appear on Data.gov
According to Department of Homeland Security statistics, there were an estimated 460,000 unauthorized immigrants in Arizona in January 2009. That statistic, making Arizona the state with the seventh largest illegal immigrant population, was often cited last week, as Gov. Jan Brewer signed the nation’s toughest law on illegal immigration.
The number comes from “Estimates of the Unauthorized Immigrant Population Residing in the United States,” a report that the DHS releases each year, but sadly does not add to Data.gov. It takes a little digging to find it, but the most recent report for January 2009 (released in ...
The National Data Catalog is Live
After several months in development, I'm happy to announce that the National Data Catalog is up and operational. This site represents months of work by Team Ruby members Luigi Montanez and David James. Since July of 2009 when we kicked off the project, they've been working hard at building a great architecture for the system, and pulling in data pointers from sites like Data.gov, The DC OCTO and Utah's Data Catalog. Presently, we, alongside our volunteers are working on importers for DataSF, Rhode Island, and New York City.
Continue readingGovernment Data Sets – Managing Expectations
US Open Government plans were released today. As part of this process, federal agencies are beginning to release data sets publicly in ways they never have before. Some substantial and thought-provoking blog posts over the last few weeks have discussed how government can do open data well.
There are significant cultural and social sticking points that have yet to be addressed in releasing data openly. A discussion with a colleague from NASA last week confirmed how far away most agencies are from the luxury of considering the innovative ideas for data set management available to them. Here's why:
Continue readingOther dangerous mines? Federal data can’t tell you
Monday’s explosion that killed 25 miners at Upper Big Branch Mine in West Virginia came at a mine that had been flagged by inspectors for a series of violations – 3000 since 1995 and more than 500 in 2009 alone.
How does that compare to other mines? Because of the way the federal government releases the data, we can't say.
Data on safety inspections is published in the Department of Labor’s Mine
Safety and Health Administration (MSHA) Web site, where users can find information
on mine safety, inspections and violations. But getting to it is the hard part ...
Drafting Guidelines for Government Data Catalogs
A major focus of the Sunlight Labs is to push government to publish its data online. In recent months, we've gained in-depth familiarity with government data catalogs through our work on the National Data Catalog. The most prominent example of a data catalog is data.gov. Since its launch last year, a handful of states and cities have followed suit with their own efforts. As more data catalogs come online, we want to make sure their contents are open and exchangeable. We want to determine how to best structure the data catalog itself, and we want to ensure that the metadata it contains -- the data about the data -- exists in the most accessible way possible.
Last week, Clay posted three challenges for the community to tackle, and this is challenge #3. We're looking to start this conversation now and move towards consensus within a few months. I was at Transparency Camp, digging deeper into this topic, putting us on the path to make recommendations that governments can adopt quickly.
Continue readingIntroducing the Cycle of Transparency
This "Cycle of Transparency" demonstrates, in one image, the specific actions and the variety of actors that need to work together to create the open, transparent government we seek.
Continue readingFree yourself from the Shackles of “High Value Data”
"High Value Datasets" is a bunk term.
When the feds introduced the term High Value Data, my immediate response here was "what the heck is 'High Value Data'?!" We quickly extracted the definition from the Open Government Directive and here it is:
"High-value information is information that can be used to increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation."
Now we've had a chance to go through and take a look at some of the datasets. Our http://reporting.sunlightfoundation.com is having a field day analyzing the data, pointing out flaws in the data and generally doing a great job of figuring out what's actually new in the datasets.
Continue reading
59063
