Tomorrow is the two year anniversary of the Open Government Directive (OGD), the signature transparency policy issued by the Obama administration on December, 2009.

The transparency issues facing the administration, both before and after the 2009 policy, fall well beyond the control of the OGD, which is, after all, an OMB memo. Nevertheless, the OGD is Obama's single broadest attempt to create transparency across the executive branch, and the most high profile attempt to live up to Obama's campaign rhethoric on transparency.

Over the last two years, Sunlight has become familiar with the inherent limitations of directives and declarations such as the OGD, as we've learned that the difference between an aspiration and a mandate can be a huge gap. Announcements about new transparency policies imagine the best possible impact, while implementation often looks to the minimum requirements.

So to mark the two year anniversary of the OGD, we decided to look at implementation of the Open Government Directive.  Since much of the OGD is written in broad, aspirational language, we decided to review how well agencies have lived up to the commitments they created for themselves in their open government plans.  The OGD required agencies to publish these plans, which were all posted and revised during 2010, and often included deadlines and goals for agencies to release data and tools.

Building on the work that OpentheGovernment.org did reviewing all the agencies' plans (we participated in that review), Sunlight has pulled out all the deadlines from the agencies' plans, and checked to see whether the goals were met.

The results are decidedly mixed.

In some cases, agencies' goals were clearly met.  Many of the datasets planned to be released are now available on data.gov, and the projects and tools that agencies described are underway.

Often, however, agencies have failed to live up to the standards that they set for themselves as a result of the Open Government Directive.

The Commerce Secretary never put up a schedule. The Office of Science and Technology Policy only put up 4 years of budget data. And the Department of Justice apparently decided that none of the data they identified for public release was fit for publication on Data.gov.  Perhaps most egregiously, the Department of Homeland Security rescinded part of their plan to post a schedule for new data to be released, and released a new version of the plan scrubbed of that milestone.

Any broad declaration or aspirational policy is going to face complex challenges, as we've noted. But the agencies applied these goals to themselves. Far too often, agencies are failing to live up to the transparency goals that they themselves created. And this review only the concrete goals that agencies set. Many agencies didn't even go through the hassle of setting detailed deadlines for themselves. The Defense Department won't show up on most of our evaluation, because they hardly set any deadlines at all, and mostly committed (doc file) to talking about openness and doing some untrackable internal reviews, despite their size and obvious public importance.

We've expressed disappointment in the Open Government Directive before, pointing out, for example, that the Open Government Directive requires:

A strategic action plan for transparency that... (3) identifies high value information not yet available and establishes a reasonable timeline for publication online in open formats with specific target dates.

Cass Sunstein (director of OIRA) once called me Oliver Wendell Holmes for insistently pointing out that agencies who said they'd make some future plans failed to live up to this requirement -- that a plan to make plans was an obfuscation, and that if the White House didn't take this seriously, they'd create the wiggle room that would let agencies evade the White House's best intentions. If he meant that I was being too much of a realist, then I stand by that assessment even more firmly now, as most agencies have clearly failed to comply with that passage.

And that cuts to the heart of the OGD. Openness without information is emptiness.  If some agencies won't even share the plans they've made for publishing new information, how far can their commitment to openness possibly go?

The Open Government Directive has caused a lot of good.  And it has also often failed to live up to its promise, the administration's rhetoric, and agencies' own self-imposed compliance plans. We should remember that Presidential rhetoric and bureaucratic commitments are not the same thing as results, especially as even more administration work happens through broad, plan-making executive actions and plans.

Transparency proclamations are valuable, but the path to transparent government runs through a thousand fights over information. The OGD may have moved the default slightly towards openness, but it doesn't win those fights alone.

We'd like to invite you to review our evaluations of agencies' progress (searching through all the agency sites and plans can be tricky), and to help us think about where the OGD should go from here.

Policy Fellow Matt Rumsey wrote this post. 

As part of its Open Government Directive, the Obama Administration took steps to make a wide variety of federal data publicly available online. The Open Government Progress Report, released in December 2009 lists open government projects and transparency milestones that support the goals of the OGD. Included among these are a number of important data-sets.

These data-sets have been utilized by journalists, bloggers, organizations, and citizens. They have informed investigative stories and think tank reports, contributed to unique and useful visualizations, inspired projects dedicated to helping the American public make better use of data, and helped to shine sunlight on previously hidden areas of government.

As part of the OGD, the Department of Treasury released IRS Statistics of Migration Data. The data tracks how tax return filers moved around the country and has helped illustrate migration patterns through space and time. The data has been well utilized by a variety of sources. Forbes took the data and created an attractive and easy to use interactive map. Nielsen found that migration from the Northeast to the South and Southwest was correlated with underwater mortgages, while Ad Age focused their analysis on the migration of people as well as money. Additionally, Brookings Institution scholar William Frey used the data as part of a report on recent shifts in migration trends.

The Occupational Safety and Health Administration began to publish employer-specific information about occupational fatalities. This data, intended to help employers identify dangerous conditions and "take steps to improve safety and prevent future accidents," has also allowed investigative journalists to expose unsafe conditions and tell heartbreaking stories. The Center for Public Integrity and Huffington Post took advantage of the OSHA data in a series of articles that expose unsafe working conditions.

Data posted on recovery.gov proved useful in a number of ways.  It allowed interested parties access to information to assess stimulus programs. Data available on the site contributed to reporting on the collapse of Solyndra, helped bloggers in Wisconsin outline benefits to their state, and allowed academics to create novel tools to analyze individual stimulus projects. The nature of recovery.gov also made it easy for critics to weigh in on the shortcomings of the tracking process itself, by identifying areas where data is missing or incomplete.

Data.gov, a central repository for Federal government information, has attracted millions of viewers and inspired major cities, several states, and even other countries to launch sites of their own. Information in the database was used to create some unique and useful applications. AnalyzeThe.US aims to "enable anyone to develop an intuitive picture of the complex flow of resources, money, and influence that affect how our government functions."  Similarly, DataMasher pulls from data.gov and other sources and allows users to compare state level data on various issues.

This post represents a small sampling of the ways that data released via the OGD has been used.  It has proved to be a valuable resource for journalists, researchers, and average citizens.

Are the PR flacks of the Obama administration against government transparency? If not, then why have some instituted media policies that effectively gag federal scientists, sometimes preventing them from informing the public via news media about federal scientific research, data and policies? (I suspect there are many government flacks who are frustrated by these policies, but must enforce them as part of their jobs.)

This week, Sunlight staff shared and discussed a damning article in the Columbia Journalism Review about a disturbing trend: federal agencies such as the Department of Health and Human Services (HHS) are establishing guidelines for interacting with the press that subvert the role of their public affairs departments. My counterparts within government are now less likely to help reporters understand scientific data; instead, they serve as gatekeepers whose primary role seems to be to thwart the public’s right to know.

At first, I thought my co-workers at Sunlight reacted with a knee-jerk disdain for public relations work because the guidelines route press calls through a centralized communications team instead of letting journalists dial government scientists directly. But, reading the CJR report, which is informed by a survey of science, health and environmental journalists conducted by CJR and ProPublica, I realized these policies aren’t about ensuring consistency of message (unless you count silence as a message) and efficiency. They were about censorship.

It’s ironic that such policies stem from the White House’s efforts to ensure scientific integrity and public communication. This memo [PDF link, which is frustratingly an image PDF] states that such policies should allow for “openness and transparency with the media and the American people.”

But that’s not what’s happening, and that’s a shame as it tarnishes the reputation of other sound public relations professionals. (There are many of us, even in DC!) There's something to be said for having a good public affairs office that routes inquiries so that the agency is coordinating its messaging for accuracy and consistency. That's what we do here at Sunlight. Sure, there may be times when a reporter gets through to our spokespeople directly, who grant the interview and then tell my team about it. But for the most part, I try to ensure Sunlight’s Communications team is the first line of contact with reporters (and others), so that we can figure out who's the right person to help, what pertinent projects/blog posts we can promote to provide better context and, finally, make sure the interview happens on time. We also keep track of press inquiries to spot trends in what topics are gaining popularity (sometimes prompting us to coordinate research and post relevant updates to our blogs) and keep on top of who's interested in specific topics (so we can pitch to them in the future).

This is not the same thing as what CJR describes is happening now in the public affairs departmemts in agencies like HHS, the Environmental Protection Agency and the Federal Drug Administration. And, as these are new media relations policies, they’re not a sign of bureaucratic business as usual. For example, CJR describes the obstructionism Felice Freyer, who chairs the Association of Health Care Journalists’ Right to Know Committee, encountered in trying to get an interview for an article in The Providence Journal, where she has covered the medical beat since 1989:

Unsure which press officer to approach, she filled out the “Timely Response E-mail Form” on the agency’s website. Several hours passed with no response, so she called and spoke with a press officer. He suggested that Freyer e-mail her questions to him, which she did. Nothing. When she called again two days later, the press officer said he was waiting for a response from his superiors. He suggested that she resubmit her questions for a third time. She did, to no effect. Several more days passed and she sent yet another e-mail asking if she could expect answers, and if not, why. “At this point, all we can say is that the FDA is continuing to look into these cases,” the press officer replied.

[There’s more I could excerpt, but I highly encourage you to read the full CJR article instead.]

Sunlight’s Reporting Group journalists also often face the same deliberate impediments by administration public affairs officials. Just as the reporters surveyed in the CJR piece, they’re left hanging when requesting interviews, and after weeks (or even months!) of persistence, told they can only cite government officials on background. Citing unnamed sources hurts the credibility of journalism, which can be further compromised if later challenged to prove the truth of such unattributed statements. Likewise, Sunlight's own experiences in trying to obtain government information using Freedom of Information Act (FOIA) requests reinforce the notion that the Obama administration has not lived up to its promise to be more transparent. So much for our ability to report and shine a light on the government in a timely fashion. (This also frustrates yours truly, who sees news cycles come and go, knowing we have a good analysis to inform the public debate that can’t be published and pitched because we can’t get the information we need out of the government.)

President Obama won the hearts and minds of many American voters with promises to open up government. His administration made initial strides to deliver on that promise, establishing policies like the Open Government Directive and making government data available online for anyone to access on Data.gov. The thing is, you can’t just report on data in a vacuum. It needs to be explained and contextualized by experts. And those experts are being muzzled by policies that reek of obstructionism and an outdated culture. These policies likely prevent the administration’s PR flacks from helping to get the word out on news that would even make the administration look good. It’s as if the public affairs departments have forgotten they serve the public interest -- let’s not forget it’s our taxes that pay for their salaries.

We can’t stand for this. If you’re in DC, I encourage you to RSVP right now for an event this coming Monday at the National Press Club that will focus on this precise issue organized by the National Press Club, CJR, the Society for Environmental Journalists, Association of Health Care Journalists and Reporters Without Borders. And, if you’re a journalist, I urge you to cover this topic. Besides the groups listed above, there are many others working to ensure scientific integrity like the Union of Concerned Scientists and government transparency. Our media policy is simple: contact our Communications Manager Liz Bartolomeo or me (gab at sunlightfoundation dot com). We pick up our phones and reply to emails promptly, promise!

This fourth of July marks the 45th anniversary America's freedom of information law. FOIA transformed our world by giving teeth to the public's right to know. It made government prove why public information shouldn't be disclosed, instead of forcing people to justify why it should be. The internet age has brought another revolution, transforming how we expect information will be available to us. But our freedom of information laws have slowly calcified through a combination of inertia, bureaucratic reluctance, and lack of funding.

Just as FOIA of the 1960s embraced disclosure upon demand, it's time to reinvent our freedom of information laws for the internet age and embrace affirmative disclosure -- where government information is routinely published online, in real time, and in machine readable formats.

We have already seen some baby-steps in this direction. For example:

• The 1996 Electronic Freedom of Information Act Amendments (or E-FOIA) required agencies to publish records that they expected would be the subject of multiple requests in an "electronic reading room," and instructed agencies to make records available in electronic format when requested.
• The Obama Administration created Data.gov, an online repository of "high value, machine readable" datasets generated by the Executive branch, and launched the Open Government Directive, which (among other things) encouraged agencies to release more information to the public. (These initiatives have been subject to significant funding cuts.)
• The Office of Government Information Services Office of Information Policy at the Department of Justice* recently launched FOIA.gov, a one-stop shop to see how well agencies are staying on top of their FOIA requests.
• Nearly all agencies have websites where limited information is disclosed about their operations and activities.

The scope of the public access is much larger than these initiatives can address. The public's information needs spans the scope of government activity, from census data to spending data, from policy and position papers to economist forecasts, from maps to legal information, and much more. It is worth highlighting four particularly clever ideas for next steps as examples of where things could go.

The Public Online Information Act (or POIA) is pending legislation, originally introduced by Rep. Israel and Sen. Tester, that would require executive branches agencies to publish all publicly available information in the Internet in a timely fashion and in user-friendly formats, and create an advisory committee to help develop government-wide Internet publication standards. Like FOIA, POIA allows the public to go to court if agencies fail to comply.

In the UK, the website WhatDoTheyKnow.com makes many government information requests publicly available. You submit your FOIA-like request through the website, they submit the request to the appropriate agency and publish the answer online. This reduces the number of duplicate requests while making the sum of information released available to everyone. (There's a similar effort to crowd-source state-level freedom of information requests in the US called MuckRock. Disclosure: Sunlight gave funding to MuckRock.)

Data quality is a big deal. The ten open data principles lay out a means to evaluate the extent to which data is open and accessible to the public. Similarly, efforts like ClearSpending mash up government data to evaluate the accuracy of information reported by the government in the first place. It's difficult for the public to make use of information that is badly formatted or inaccurately reported.

It's not enough to make sure that the data reported by the government is accurate. We must also ensure that government identifies and reports on all the data that it has. The government must audit its holdings and build an index of what it has and who is responsible for maintaining the information. The President's recent Memo on Regulatory Compliance, for example, addresses how private entities disclose information to the agencies that regulate them and how that data is reviewed, shared inside government, and (whenever appropriate) made available to the public.

The public's need to access government-held information is as old as our libraries, post offices, and government publications. It's time to make our public information available online.

• Corrected to reflect the right agency. OGIS mediates FOIA disputes and reviews agency compliance with FOI; OIP launched FOIA.gov.

On Thursday, the White House announced the winners of their Leading Practices initiative, that they first outlined in April.

The Leading Practices were designed to highlight examples where agencies have risen above the expectations set by the White House, and proactively attained a higher standard of transparency. (I participated in helping to establish the leading practices standards.)

The winners are a collection of some of the best transparency work being done at federal agencies, with HHS taking the slot for transparency (quite deservingly). These winners are a nice counterpart to the White House page on Open Government Highlights.

As I wrote when the Leading Practices were first announced, though, there is a bittersweet element to this congratulatory platform. As the White House rightly points to the great work some agencies are undertaking, we can't help but wonder whether there is an analagous effort being undertaken with agencies who are struggling with (or blowing off) the Directive's requirements.

While we can hardly expect the White House or OMB to publicly chastise any laggard agencies, we do have to wonder how much of a private stick exists to go along with this public carrot.

The New York Times this morning took the Administration to task because White House officials and lobbyists are meeting at a local coffee shop, undermining the White House transparency rules for meetings between its officials and lobbyists. We're not naive and didn't expect that releasing the Visitor Logs would capture all meetings between lobbyists and White House officials. I’ve been a Washington denizen for too many years to believe that we’d have total transparency of a very protected relationship.

But the story gives me an opportunity to think about a nagging and ongoing concern: are we being lulled by the rhetoric of transparency?

I have believed -- and still want to believe --  that the Administration is sincere in their efforts. It’s not a stretch – but maybe not much of an accomplishment either – to call this the most transparent Administration in (recent) history. No other Administration has even talked their talk, much less tried to walk their walk.

But where there’s a will to hide something, there’s always a way. What concerns me now is whether all the White House promises are anything more than “transparency theater.”

On the one hand, there have been significant pronouncements about transparency for lobbyist activity, and about the need for open information and data. The Administration, from the first full day in office, has made this at least a rhetorical priority. “Good start" we’ve said for some 18 months. And the early signs were promising.

But  the delivery of the rhetoric has been deeply disappointing. The torrent of data we expected to see at Data.gov isn’t materializing. In fact, Luigi in Sunlight Labs quickly calculated that literally only 1 percent of Data.gov is useful if you want to do something other than build a map. (That is, 99 percent of the 272,677 data listings are GIS data -- shapefiles that are only useful for mapping software and similar tools.)

USASpending,gov data is riddled with so many errors that it is practically useless. Recovery.gov has been consistently rushed and is having a hard time finding the right balance between flashy presentation and substantive data. And the Open Government Directive delivered very little data and offered few substantive plans for either agency audits or release of it.

Call it a hot Friday afternoon, and maybe I'm cranky, but I think it’s time to begin to ask some tough questions of the White House. Whither or wither transparency?

Cabinet agencies (and others) released their Open Government Plans last week with much fanfare, mixed reviews, and many promises for the future. I want to focus on one initiative -- the Department of Labor's "Online Enforcement Database" -- to highlight the strengths and weakness of what we've seen, and suggest some guidelines for going forward.

Online Database Strengths and Weaknesses

With the explosion at a mine in West Virginia last week, many questions are being asked about federal safety inspections. My colleague Anu Narayanswamy wrote on Monday, before the Online Enforcement Database was released, that the way the federal government releases data on mine safety makes it impossible to see how safety violations at one mine stack up against others. You cannot tell if the 500 safety violations in 2009 at this particular mine, for example, are typical for this industry.

On Wednesday, the Labor Department released the Online Enforcement Database, which contains five major data sets, including one on mine safety. Anu's follow-up article on Friday explained that "with mine safety data, released for the the first time in bulk [on Wednesday], users can search for mine inspection data by state or even zip code." But she also reported the data sets are only in a partially downloadable format, and do not include "the kinds of violation and penalties levied on mines across the country." In other words, it's difficult to figure out what's going on.

It is the search results, and not the underlying database, that are downloadable in bulk. ("Bulk" access means that you can download all of the information at once, and not piecemeal.) The only way to get at the Enforcement Database's information is to use its search tool, which has very limited capabilities. Users may search by state, agency, zip code, and by industry code. (DOL deserves credit for including the industry codes in a link from the search page.) So, a user cannot narrow the search range to a county, or a congressional district, or by the owner of a facility. Compare this to the search tool used at transparencydata.com, a new initiative from Sunlight that allows users to search a database on campaign contributions, that allows searching, sorting, and downloading in a multiplicity of ways.

As mentioned before, the Online Enforcement Database itself is not available for download in bulk. There's no way to look at all of the information the Labor Department has painstakingly gathered. And despite the wealth of information, a clunky search tool adds to the frustration. Without access to the supporting data, researchers cannot answer many questions. In fairness, the Labor Department says that bulk access and improved search tools are "coming soon," but it would be very helpful to have a date to accompany this promise. Doing so would make the promise concrete and testable.

I do not mean to pick on the Department of Labor, which made an effort in its Open Government Plan [PDF] to identify datasets for online publication and to set deadlines. Indeed, they stated they plan to take all data they collect and make it publicly available online and in downloadable formats, with appropriate caveats. Many agencies fell far short of DOL's achievements. But DOL should go further.

Open Data Principles

Elsewhere I've pulled together resources (from Princeton and Sunlight Labs) on building good data sets, including drafting guidelines for government data catalogs. It's important focus, however, at the fundamental level of what it means when we talk about how government should publish data online, a.k.a. "open data principles." As an attorney, I'm hardly qualified to talk about this, so I am fortunate that much of the heavy lifting was done at a conference in 2007. Afterward, my colleagues Clay and John and I worked on revising the open data principles, nine in number, and fleshed out a rough evaluation of when they are satisfied.

When agencies think about how to make information available, they should look to these (draft) principles. They state, in short, that data should be: complete, primary, timely, accessible, machine processable, non discriminatory, non propriety, license free, and permanent. Resource to these principles by the agencies -- and a better effort to comply with the directive's requirement to identify all high-value data sets and set deadlines for online publication -- would have turned the thus-far mixed results of the Open Government Directive into an unqualified success. There is still time to make that promise into a reality.

Here are the 9 open data principles in a framework to evaluate the extent to which they are satisfied:

(If you're wondering what ^M means, it's old school geek for delete - the 8 principles of open data have now become 9.)