Last week, the Office of Management and Budget (OMB) posted updated guidance to the Project Open Data Github for agencies as they continue to compile indexes of their data holdings. Agencies are now instructed to “Include all ‘non-public’ data assets in their PDL [Public Data Listing], in addition to the ‘public’ and ‘restricted’ data assets that have long been required.” It also tells agencies to fully explain the reasons for “non-public” designations and redactions in the metadata.

This change turns the one-time results of a Sunlight FOIA request into effective policy moving forward, ensuring that the public will continue to have the fullest possible knowledge of agency data holdings. We have been urging OMB to embrace this method since before our FOIA was fulfilled — and we’re thrilled to see it actually happening.

The guidance builds on last month’s historic release of comprehensive data indexes (referred to as Enterprise Data Inventories or EDIs) that was sparked by a Sunlight Foundation FOIA request.

The change validates our belief that releasing full EDI’s — with appropriate redactions under FOIA — would not prove to be unnecessarily onerous on agencies.

As of today, the majority of agencies have moved to comply with our FOIA by releasing appropriately redacted EDIs. Future compliance should be as simple as a quick check with an agency FOIA office as new data assets are identified and included in Public Data Listings, though we understand that this initial change has and will require somewhat more effort.

Most agencies seem to have accepted and understood the need for this change, but some concerns have been voiced:

• The Department of Commerce, which holds a very sizable amount of data, has cited “certain technological issues that are creating major barriers to creating the PDL as described [in OMB’s guidance].” This is a concern, but Commerce appears to be working through these issues with relevant officials in the General Services Administration and OMB.
• NASA has expressed significant concerns with the new process, specifically citing pushback and time constraints from its FOIA office and legal team. They claim that “identifying and indexing our open data is our priority,” and indexing its non-public data will take valuable capacity away. Fortunately, NASA has bought into the public process and are engaged in meaningful conversation about the agency’s concerns on the Project Open Data Github.

We’re looking into these concerns and we’ll have updates moving forward.

Overall, this is a big step toward systematic openness and should allow interested stakeholders to more fully engage with government data assets on a consistent basis moving forward. We’ve already heard from journalists who are working to parse and understand these data.

We are happy that our efforts showed OMB (and government agencies) how easy openness can be. We are hopeful that this will result in more government data being opened — and used — in the long run, and that it will enable FOIA requesters to better target their records requests, in turn providing better results and limiting the need to search for responsive records within agencies.