Open Government: idling in the driveway


Sigh. I feel like a disappointed parent.

When the details of the Open Government Directive were announced early last December I was unbelievably excited. Seriously. My long time hope that one day government would get “it” about the importance of putting public information online appeared to have arrived. Government data was going to become available as a default and that was going to start with an “inventory” (government’s word) of the “high value information” (also their words, though less than ideal because who would ever agree what that means?).

Agencies were supposed to do two things with respect to releasing data: create an inventory of the “high-value information” currently available for download and identify high value information not yet available along with establishing a reasonable timeline for publication of that data online. It was that latter requirement that I salivated over. Certainly there are other important aspects of Open Government — participation and collaboration are values we hold dear at the Sunlight Foundation. Car IdleBut yesterday was the day when the rubber was supposed to hit the road on data. For many agencies, they didn’t even get out of the garage.

There are some very interesting data that’s going to made available, almost immediately (and John Wonderlich, our Policy Director, has a post on it) but some agencies avoided the requirement entirely, some decided to say they’d make a plan to plan how to identify and release data, and others mentioned it but didn’t explain how they would achieve it.

First, our quick review shows that a little more than half of the 30 agencies’ plans we reviewed (18) specifically identified new data to be released — 12 did not. (This includes some independent agencies.) The total number of data sets identified to be released — approximately 89.*

89 data sets identified for release – across the entire federal government!? I’m speechless. I was looking for inventories of data (this is the Directive’s word, after all) — actual audits of what data each agency collects and dates of when new information would be made available. That is not what we got.

The Department of Health and Human Services (HHS) was among the best – identifying 14 new data sets to be released – and this is crucial data. While maintaining the privacy and identity of patients HHS will be releasing critical data about Medicare: everything from inpatient hospital stats to prescription drugs and hospice care. During an era on increased responsibilities for HHS this data is absolutely critical to keeping HHS effective and accountable.

Few agencies rose to the high water mark of HHS. Part of the problem might be attributable to cultural barriers and the illusion that some bureaucrats hold that this is “their” data vs “all of our” data. Part of the problem might have been time to pull the information together.

Maybe, a bigger part of this problem is a loophole in the Open Government Directive itself. By asking agencies to only inventory their “high-value” data it gave them an instant out for just about anything. Despite the White House’s good intention in defining high-value as: “increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation.”

With a definition like that “high-value” could mean literally anything: if you collect a piece of data that is not to “further the core mission of the agency” why are you collecting it?

When you define a concept too broadly you end up not defining it at all. If we could roll back the clock on the Open Government Directive we would ask agencies to first list all data they collect and then create sub-lists of:

  • data that is currently public but not online
  • data that is currently public and online
  • data that is not currently online but that will be put online and when
  • for everything else, explain why it won’t be put online

This would give us an instant picture of what the online (and therefore, public) landscape of federal government looks like and is an invaluable data set in its own right.

HHS, NASA, Education, National Archives and Records Administration and the Office of Personnel Management were the high water marks.

Defense, Homeland Security, Justice, State, Interior, Treasury, Veterans Affairs, US Agency for International Development and the Social Security Administration did not identify any new data to be made available – no inventories either.

Yes, I appreciate the extraordinary hard work put into the Open Government Directive by all those in the agencies and those spearheading it at Office of Management and Budget and the White House, and I wouldn’t suggest that evaluating these plans based on just one of a couple dozen appropriate criteria is a totally fair reading of how successful this exercise was, but I have to look at it from what I feel is key for government accountability – data. That’s my lens on the world.

We’ll continue to evaluate agency plans all next week.


*We arrived at the 89 number via a very generous methodology. It all depends on how you define a “data set”. Our complete inventory using a more exact methodology will be available soon.

Photo credit: “Idling” by Flickr user N1NJ4.

Categorized in:
Share This:
  • Have to fight the distorted data with counter true data.

    Checkout, the distortion possible in the IT Budget nearing $80 Billion.


    So many of them have wrongly classified data created over the years, as if Fascists were running the office with a carefully planned and masterfully orchestrated administration, who made everything look so awesome. Obfuscation can be defeated by forensic abstraction.

    The Financial reform has to begin in DC. A simple audit will reveal much toxic asset.

  • PS

    In the past 30 years, I have found myself working as an in-house auditor, or as an independant auditor, or the only guy in-house who was willing/able to work with the outside auditors despite the inevitable: scapegoat outcome… $$$RULE, distraction is provided per the P. T. Barnum quote: “One born every minute” and those born as such are ALL invited to the TEA-PARTY hosted by a cheerleader from Alaska who plays the MAD HATTER way better than J. Deppe, or OUR former-male cheerleader Prez ever could…

    Hat’s off to Carl Rove: Who Knew that placing the future of the entire NATION would make a shitload of $b$u$c$k$s$ for a few at the expence of everyone else…

    The TEA-PARTY is good, but they are setting up the Gullotine for the wrong heads, and the folk (us: the US) are so distracted from anything even resembling a fact, that, well, we all might have to die during the last ‘musical scene’ few minutes of that “Dr. StrangeLove” movie because THEY THINK JESUS will fix this mess they made…

  • Eric

    Kevin, in FY2010 the Feds will spend $706.4 billion on Defense and HHS. Half of that is $353.2 billion, not $500 billion, so you’re off by about one-third.

    Also, the biggest item in the budget is Social Security, at $695 billion. Social Security, Medicare, and Medicaid combined are $1.44 trillion, more than twice as much as it costs to defend the country.

    These numbers are available from open datasets — you should try reading them sometime, although you’ll need something better than liberal math to interpret them.


  • Mary,

    I agree. We should keep the government small.

    That is why I am sure you will not object to my suggestion of immediately cutting in half the budget for the largest piece of our U.S. government, the U.S. Military and Homeland Security.

    Wow! That’s a savings of over $500 billion per year! Hey, what do you know! Instant small government! Problem solved!

  • Gerald McCollam

    Sharon writes:

    “Are we certain there is ‘high-value’ government data in existence to inventory? ”

    I think it’s hard to gauge whether ‘data’ intrinsically has value separate from the potential uses it might be put to. Data in itself is quite meaning-less.

    A counter example might be some unknown (or unknowable) data that’s held against someone else as a kind of threat, sort of the way J. Edgar Hoover is said to have intimidated his enemies. Hoover could unnerve simply by indicating he had ‘potentially revealing data.’

    While data used in this fashion has some intrinsic meaning, data has little value – low or high – simply because it’s being held in a government database. And this is precisely the point for making that data more accessible – so that it may gain value through meaningful use.

    This reminds me of an old epistemology course: if I take a single barometer reading today, I have data. If I have a reading from yesterday and one from today I have information – I know if the pressure is rising, falling, or unchanged. If a series of such readings leads me to recognize that lower barometric pressure often leads to poor weather, I have knowledge. And if, knowing all I do about the weather, I know I can still be wrong about it, I have wisdom.

    Lesson: we’ll never get to wisdom without first getting the data!

  • The biggest problem in our current state is that it most of our politicians are lawyers which allow them the gaul to pass legislation that the politicians required to pass it don’t read it for the benefit of the non-lawyer politicians. for the status-quo lawyer types the bills are worded so as to allow loopholes in be cloaked in gobble-de-gook in my opinion.
    That is where accountablity should begin. Once accountability, by that I mean that proposed legislation has to be fully understood by society before it can be voted on.
    Transparancy is the next logical step in this process. After a piece is logically and re-interpreted in non-legalese and fully understood by the citizens can it be vetted. No doubt that is how so much government waste has crept in to the mix leading to huge deficits and an overbearing tax burden.
    I want to do anything I can to further your cause. We have to make this republic work the common good and not for the furtherance of political agendas.

  • Mary

    Please do not be concerned about the specifity of your “high priority” request to the government agencies.

    I used to work for a government agency and I found that no matter how thorough and specific the request they will stall, give excuses and come up with a unending supply of reasons they will not be able to comply.

    That is the reason our government is so ineffective and has become so large they are violating our rights and they believe they have a right to do so.

    It is also the reason government absolutely has to be kept small.

  • screwed-taxpayer

    And lost in all this noise is the egregious, and likely unlawful/malfeasant practice of gov-employees signing away exclusive-rights to academic papers they publish (to for-profit and obscenely expensive academic journals).

    I ask those who are fighting this fight for us to PLEASE add the following to your priority-list:

    Every single paper authored by a public-paid employee, whether directly in gov, or in taxpayer-funded universities, be made freely available in full-text online…specifically including all past papers by these people as well!

    thank you

  • Have to agree with Andrea and A Govvie. The problem is the requirement itself. It is so vague to be almost meaningless. The big agencies have tons of data. To comply, all they had to do was to pick what was easiest to release and somewhat useful. Even to do this, however, was probably difficult for the poor fed (or, more likely, contractor) who ended up with this task, after it was handed down, layer after layer. He or she would have to do this, in addition to all of their other duties, without much guidance or support. It’s an unfunded mandate, like many that are handed down from on high – new rules on credit card use, recycling, conference room use, travel, IT security, new badges, foreign visitors and, oh yes, data transparency.

    Government should be more transparent but the directive seems more like a PR exercise than serious change.

  • Andrea Schneider

    I am not surprised at this disappointment, I would have expected it for several reasons and underlying organizational causes.

    1. Did we give agencies the tools, education and clarity of expectations across the board? Did we define our terms clearly enough, so everyone was playing with the same playbook? Probably not. I don’t think we’ve done that for any of the stakeholders, including the public.

    2. Did we do everything we could to help them do the “right thing” and provide support and help along the way?

    3. I’m sure we crashed up against intransigent internal leadership with deep groves in the earth, who know how to do just enough to get by. Did the agencies even have the information requested? I doubt it in some places.

    4. Did we ask that all the agencies act in accordance with the values we are creating? Or was this just another expected report, with “we’ll see what happens, we know how to outlast any elected administration”.

    I have a hard time believing there has been enough corresponding organizational change to support really excellent plans. I’m curious what tools we used to help build organizational capacity to deliver what we really wanted or expected.

    I suggest we take a good look at past efforts, this effort and look hard at what we know will create success and use it. I think we will be less disappointed next time.


  • Sharon

    This sentence made me giggle:

    “By asking agencies to only inventory their ‘high-value’ data it gave them an instant out for just about anything.”

    As in: Just about anything they have in their inventory is not considered by them to be data of “high-value”?

    As in: If you asked me to turn over my inventory of everything I keep on my yacht, I would have an instant out not to comply. I don’t own a yacht. Therefore I couldn’t turn over what doesn’t exist.

    Are we certain there is ‘high-value’ government data in existance to inventory?

    Maybe you should have asked for an inventory of their work they consider low value. Might have gotten their whole darn data bank.

  • Maureen

    I worked at a hospital (non-clinical staff) for a while. I understand the emphasis on privacy and security of data. (Although my feelings on data privacy and insurance companies would make me go off on a rant, so we’ll skip those.) It is important to bear those privacy concerns in mind. But one administrative person at the hospital was fond of saying, “There’s a lot of people who get away with not doing anything because of their (air quotes here) concern for privacy and data security.”

  • Willey

    What happens when the information is posted and is incorrect. What were to happen if “Highly Sensitive Information” such as personal identifiers was released and that information was also incorrect. Who would be accountable for this information and how would one be able to correct the information that was being viewed and acted upon. What sensible measures would be taken for accuracy? There are problems with opening up to the electronic data age and not enough people who are knowledgeable in the industry to hold the government accountable. When reviewing such organizations such as the Electronic Privacy Information Center can open the eyes of some skeptics like myself. Sensibly it is a great idea to make it easier for people to read the bills and resolutions of the laws that our governments (including Local & State levels) use to guide we the people of the United States of America.
    With this new technological age of dissemination of data come the great responsibility of accountability.

  • Greg

    In my experience as an IT contractor for the federal government, getting agencies to share data is very difficult. They typically hide behind the term security rather than figure out how they can share the parts of the data that are not a security of PII risk.

    If it takes years of negotiations, signed memorandums between agencies and SFTP connections to share the data internally I can only imagine the cultural and technical barriers to sharing the data on the open internet.

    That’s not to say it couldn’t/shouldn’t be done. But despite all the promises of transparency, this administration doesn’t appear to be spending much effort on managing its 2 million plus workforce.

  • A Govvie

    You assume that the government employee that these plans landed on to write has nothing else to do. When in fact they are probably the “Go to person” for their entire agency and have a pile of crap on their desk 24/7 because everyone else in their agency is either too dumb or too Bureaucratic to ever ACHIEVE anything.

    Now, if the Administration was really serious about this, they would make agencies form funded offices of “data and transparency” that would govern these releases. But, NO… they decided to make it yet another unfunded mandate. Hey how is that Limited English Proficiency executive order from CLINTON doing these days???? You see, without funding these mandates are worthless.

  • And these are the big agencies sitting idle.

    My local county government doesn’t even keep an internal log of all its systems, let alone specific datasets to make public.

    Speaking to local elected officials about transparency: I’ve found there is a personal buy-in to the notion of transparency, but a political hesitancy and misunderstanding of what is actually involved – “What does transparency look like?”

    I’ve found it helpful to de-emphasize the burden (time & effort) of transparency by making it clear that their role, for now is minimal. Step 1 can go a long way, just publish data.

    We, the public, can do Step 2. We’ll format it, provide context, make it ‘pretty’ and understandable.

    And together, WE, citizens and government, can work to realize Step 3: a more engaged process. (Though 2 and 3 overlap considerably).

    For those of us looking forward, it’s tough to see the possibilities ahead and not to be able realize them as quickly as we could through technology alone, because its the gov culture that resists and lags.

    Nevertheless, the fact that there is a federal directive with “Open Government” in the title is a huge step, relatively.

  • Step 1) publish data in open, machine-readable forms

    looks like a small first step so far ;)

    Step 2) aggregate & map various forms of open datasets (web, streams,enterprise) into useful open semantic frameworks

    Step 3) bring more citizen engagement into the process, to invite the public into a dialog about and contribute to the data to in order to shape policy.