For the first time, the United States government has agreed to release what we believe to be the largest index of government data in the world.

On Friday, the Sunlight Foundation received a letter from the Office of Management and Budget (OMB) outlining how they plan to comply with our FOIA request from December 2013 for agency Enterprise Data Inventories. EDIs are comprehensive lists of a federal agency’s information holdings, providing an unprecedented view into data held internally across the government. Our FOIA request was submitted 14 months ago.

These lists of the government’s data were not public, however, until now. More than a year after Sunlight’s FOIA request and with a lawsuit initiated by Sunlight about to be filed, we’re finally going to see what data the government holds.

Sunlight’s FOIA request built on President Obama’s Open Data Executive Order, which first required agency-wide data indexes to be built and maintained. According to implementation guidance prepared in response to the executive order, Enterprise Data Inventories are intended to help agencies “develop a clear and comprehensive understanding of what data assets they possess” by accounting “for all data assets created or collected by the agency.”

At the time, we argued that “without seeing the entire EDIs, it is impossible for the public to know what data is being collected and stored by the government and to debate whether or not that data should be made public.”

When OMB initially responded to our request, it didn’t cite an exemption to FOIA. Instead, OMB directed us to approach each agency individually for its EDIs. This, despite the fact that the agencies are required to submit their updated EDIs to OMB on a quarterly basis.

With that in mind, and with the help of some very talented lawyers from the firm of Garvey Schubert Barer, we filed an administrative appeal with OMB and prepared for court. We were ready to fight for the idea that government data cannot be leveraged to its fullest if the public only knows about a fraction of it.

We hoped that OMB would recognize that open data is worth the work it takes to disclose the indexes. We’re pleased to say that our hope looks like it is becoming reality.

Since 2013, federal agencies have been required to construct a list of all of their major data sets, subject only to a few exceptions detailed in President Obama’s executive order as well as some information exempted from disclosure under the FOIA.

Having access to a detailed index of agencies’ data is a key step in aiding the use and utility of government data. By publicly describing almost all data the government has in an index, the Enterprise Data Inventories should empower IT management, FOIA requestors and oversight — by government officials and citizens alike.

The latest iterations of the EDIs are due on Feb. 28, and OMB has said it is prepared to release them to us as soon as three days after they are submitted by the agencies. Agencies will be making and explaining any necessary redactions pursuant to FOIA exemptions. (For the full details, see the letter below.)

We view this release as a significant victory for the open government and open data movement. Creating a more complete picture of the government’s data holdings is a longstanding priority for the Sunlight Foundation, and is broadly considered a big step toward open data policies that transform how governments work.

There’s no guarantee that agencies’ indexes will contain all the data that it should (because they are iterative, living documents, many of which are still in various stages of construction). Still, this announcement makes a huge step forward: Rather than wondering what data the government has, we are all now in the position of policing how completely agencies are indexing their data, deciding what to publish and determining why some data cannot be public.

Data.gov helped create the norm that public data should be centrally listed, and the new expectation — representing a significant cultural shift within government — is that all major data sets should be acknowledged and described publicly (subject to a few exceptions).

This move has built on the momentum of President Obama’s various open data policies throughout his presidency, sustained by a growing core of like-minded staff who are helping transform the government from the inside. (Like the Department of Transportation, which has long led on this effort, and is the only agency that currently publishes its EDI.) OMB deserves significant credit for acknowledging the value of data indexes, working with agencies to develop them and, now, releasing them publicly.

We are happy to say that, despite extensive delays and the need to consider additional legal actions, the FOIA process appears to have worked in this case. We hope that this release will show the federal government that it can be open about all of the data it holds, inspiring them to continue releasing EDIs as a matter of policy moving forward. It’s a very good day for open data!

—–

OMB’s letter to the Sunlight Foundation describing how they intend to comply with our FOIA request — and subsequent appeal — for agency Enterprise Data Inventories created under President Obama’s 2013 open data executive order.

OMB Letter to Sunlight Foundation 2/6/2015 by Sunlight Foundation

Our appeal to OMB’s initial response to Sunlight’s FOIA for EDIs, prepared by lawyers at Garvey Schubert Barer.