There have been lots of conversations recently -- most of them provocative in the good sense of that word -- about the success or failure of the open data and/or open government movements. I have just a few thoughts to add that I hope amplify Sunlight's position.

Sunlight believes in open data and open government not because these are abstract goods, but because we want to make government more accountable to ordinary people and less subservient to well-connected special interests. We think it's great that more consumer-facing data will be opened up by the Obama administration (aka "smart disclosure"), and we want the "operating system" of government open and free, along with many others. And to be sure, there are many additional benefits to be had from opening up government data including increasing efficiency, reducing waste, creating new business opportunities and empowering consumers.

But we remain insistent that a central if not the core goal of the transparency movement must be to shift power from the few to the many, by making all the information about who is trying to influence the process and what they get out the other end more accessible to all. That's why we keep a large part of our attention focused on opening up the political influence arena and exposing the lobbying culture, and that's why we called out (back in September 2010) the inadequacies of the Obama administration's implementation of its open government directive; why we criticized the extra-governmental crackdown on WikiLeaks; and why we will continue to press both sides of the aisle and the regulatory agencies to force open the exploding world of "Dark Money" super PACs being employed by Republican and Democratic operatives alike.

I've been at these fights a few decades now, and I have never been more optimistic. The culture of transparency as an instrument of accountability -- by citizens and government alike -- is now generally accepted. The strategy of pushing and pulling Washington -- and every state capitol and every government in the world -- will be done by the tens of millions of people online demanding answers to their questions and who will, eventually, vote based on the answers they receive or don't. Information is the key to action.

Other than his inconveniently similar name, I usually have no beef with Tom Slee. His blog provides a reliably interesting perspective on how information technology is and isn’t changing democracy. But I think that yesterday’s post (the provocatively-titled “Why the ‘Open Data Movement’ is a Joke”) suffers from a lack of perspective. And, since other smart people appear ready to accept its premises and conclusions, it seems worthwhile to respond to it in detail.

What Do We Stand For?

Tom’s critique begins by noting the Canadian government’s failure to stand behind the work of its scientists, and its decision to eliminate a valuable type of census data. The open data movement, Slee says, has not done much to oppose these developments.

I’ll admit that Sunlight’s attention does not often stray as far north as Ottawa, and protecting science from politics has never been a core part of our mission (though we often find ourselves at the table with people who work on the issue). But here in the U.S. the open data movement absolutely has engaged in activism to protect valuable and threatened information. Our Save The Data Campaign is a good example of this, as is the push (led primarily by the library sciences community) to save the Statistical Abstract.

Open Data and the Private Sector

But the core of Tom’s complaint isn’t about episodic failures of activism. Rather, he seems to be bothered by open data enthusiasts’ adoption of language and an aesthetic that traditionally belong to projects with more expressly political (and progressive) aims. He seems suspicious that a self-described nonpartisan activist movement could be anything but a cynical lobbying ploy for private interests. Indeed, there’s a clear strain of hostility toward business that runs through Tom’s critique. Fair enough: more than a few such “movements” have turned out to be astroturfing operations, and it’s certainly true that on some open data issues I expect a less than enthusiastic response from the corporate world.

But I think it’s flatly wrong to consider private actors’ interest in public data to be uniformly problematic. We should be clear: we won’t tolerate those interests’ occasional attempts to lock public data into exclusive monopolies. I think our community has done a pretty good job lately of identifying such situations and stopping them, and of course people like Carl Malamud have been doing important work on this question since well before most of us ever heard of "open data." But if commercial activity is enabled by data, that’s all to the good—the great thing about digital information is that scarcity doesn’t have to be a concern. Google Maps’ uses of Census TIGER data, for instance, is proprietary, motivated by profit, and unquestionably a huge boon to human welfare. And the source data remains free for anyone else to use! Cutting off those kinds of uses with noncommercial licensing would be nothing more than a destructive act of pique.

The Big Tent

It is true that some people are primarily interested in open data for its commercial potential. I’m not one of them, particularly. But I’m very glad to have them on my side as I seek data that can enable oversight and accountability. This tension is thoughtfully discussed in Harlan Yu and David Robinson’s recent paper, and was one of the subjects covered in a thought-provoking session moderated by Yu, Josh Tauberer and Justin Grimes at last weekend’s Transparency Camp.

The paper discusses different dimensions of the open government movement, which I would characterize as “open-as-in-data.gov” and “open-as-in-FOIA”. It’s certainly possible to separate these strains of thought, but, as I argued in Saturday’s conversation, doing so is likely to be politically counterproductive. Consider the case of the free software movement. Can there be any question that Richard Stallman’s radical political project has benefited from consumers’ pragmatically-minded enthusiasm for downloading Firefox? Stallman would be the first to explain the difference between his efforts and the less ideologically strident open source movement, but it seems obvious to me that our software is freer today than it would be in the absence of that useful ambiguity.

Open data’s big-tent strategy naturally means gravitating toward noncontroversial points of agreement (everyone’s in favor of weather data!). This can admittedly make the public face of the movement seem a bit too eager-to-please—perhaps even to the point of toothlessness. For Sunlight, this is a constant question: if the government releases a new site that simply reshuffles some existing datasets into a new presentation, should we raise hell? Or should we applaud their embrace of openness in the hopes that it will help to define future attitudes and expectations? That’s a political question, and one that we answer in different ways at different times.

Working on our short documentary series called OpenGov Champions I get to go and hang out with some extremely smart, creative and inspiring people. The Champions are ordinary citizens who do some extraordinary work to open up their local government data in big or small ways, not because someone asked them to, but because they are either fed up with not having access to information they need or simply because want their communities to flourish. To me, this is far more interesting as a storyteller and documentary filmmaker than interviewing seasoned politicians, spokespeople or experts. These are the kind of stories that are the fodder of classic storytelling and moviemaking. They are real life stories of the quintessential American heroes: Ones who defeat the odds by taking the matters into their own hands and create real change by strong will, passion and hard work.

Working on these mini documentaries is the favorite part of my job. The Champions open up their homes to me and I get to spend a few hours interviewing and getting to know them, and then a week or so putting the video together. As I watch the raw footage over and over again, looking for core of the story, I feel like I really get to know them quite well. And I always have a hard time with the inevitable elimination of footage, (called “killing your darlings” in editor talk.) They always say so many interesting things with insight, humor and wisdom that I would love everyone to hear. Yet I need to cut a lot out to get to the heart of the story and tell it in about three minutes.

I was first a little nervous calling Laura Amico to talk about doing a video about her. I had heard about Homicide Watch DC, and the imagery it had sprung in my mind was of some hardened-by-life, don’t-mess-with-me reporter straight from a film noir movie. You’d have to be to handle all that horror and heartbreak associated with homicide reporting, right? But I was relieved to find that she was none of that. To the contrary. When my coworker Kevin and I went to interview Laura and her husband Chris, who also works on the site, their warm and tranquil apartment smelled of something delicious cooking. They were a delight to work with. We stayed for more than three hours and yet they gracefully kept talking to us despite it getting late and their dinner delayed. In the course of the interview she explained that even though it is hard work sometimes, what makes it worth it is that they have been able to create this place where families and friends, teachers and co workers of victims -- and suspects for that matter -- can find information they need and support each other through the tough times. I find it remarkable that she can keep on doing this work without becoming the toughened reporter I imagined her to be in the process.

The story of Homicide Watch shows how open data and government transparency touch upon a wide variety of issues and affect so many different groups of people. Be it environmental, political, civil rights related, social, or any other small or big cause, it will likely at some point benefit from open and easy access to government data. In Laura’s case, it is violent crime data from the courts and police departments in DC.

I am not really a wonk. But I do care about transparency and openness in our government. Talking to the OpenGov Champions to me speaks more about the OpenGov movement than the more technological or political chatter you hear so much here in D.C. What I, and many others like me need in order to “get it” is a story, a human face that can connect the dots for us and show us what others do and that we can do that too. And I believe most of us need inspiration from others from time to time. Meeting the people who participate in the OpenGov movement in their own ways makes it real to me and makes me feel I'm part of a larger community.

It gives me hope to see that I don’t have to run for office or know the secret handshake in order to make change. Ordinary people coming together are what it takes -- sharing stories like this one and using them to build support for real change -- for transparency and openness in all our communities. That’s why we started this series, and I hope you all can gather around this modern version of the campfire and enjoy the story.

Open data policies aren't doing nearly as much good as they can, because they usually fail to require new information to be disclosed.  To fix this, governments should make their information policy decisions as publicly as possible, indexing their major information holdings, and publicly determining whether or not to release information.

Most newly implemented open data policies, much like the Open Government Directive, are announced along alongside a package of newly released datasets, and often new data portals, like Data.gov. In a sense, these pieces have become the standard parts of the government data transparency structure.  There's a policy that says data should generally be open and usefully released, a central site for accessing it, some set of new data, and perhaps a few apps that demonstrate the data's value.

Unfortunately, this is not the anatomy of an open government.  Instead, this is the anatomy of the popular open government data initiatives that are currently in favor. Governments have learned to say that data will be open, provide a place to find it, release some selected datasets, and point to its reuse.

What gets left out of these initiatives, however, is often the most important part -- the decisions as to what gets released, and how.  Many open government data discussions skip over the question of whether governments are deciding appropriately what gets released and what doesn't.  Instead of making complex decisions about what should be released, central governments suggest that those decisions are hard, and that as long as there's always some new information, then we're making progress that deserves praise.

Progress or not, open data policies often pretend to be something they aren't.  The Open Government Directive is simply dismissed or ignored by agencies who decide not to release information, as we've often pointed out before.

In the face of this shortfall, we at Sunlight have tried to focus on real decisions about actual datasets, and to force agencies to do the same. While that's proved difficult to do, as existing requirements like the Paperwork Reduction Act, the OGD, and the Presidential Memorandum on Regulatory Compliance are often ignored, agencies do respond when pushed on substantive, particular issues.

So we're not giving up on forcing agencies to make information policy decisions in public. One of the most important things that governments can do to be more transparent is to list, or index all of their information holdings online.  CIOs should be more than just technology purchasers; the word information is in their title. Every agency should have a public list of its major information holdings, along with a description of whether it's public or not, and why. Without creating such a list, how do Chief Information Officers even do their jobs?

Now, the question "where is all of our information" can be a tricky one to answer, but agencies can rely on threshold definitions.  For example, any database with a maintenance cost over a certain number should be listed.  Any information specifically described in a statute governing the agency should be described.  Any form, report, or data described in the regulations governing the agency should be described.  Whether the information is usually (or never) accessible via FOI request should be noted, and whether bulk data is available through a central portal should be spelled out as well. (By far, the best example of such a review that we've seen is the DOT regulatory compliance plan, and the closest we've found for Congress is this.)

If the public, and if the Congress (or other legislatures) are to be involved in creating a more open government, we need to be able to measure openness against a background that makes sense.  Governments ask to be measured against the failures of the past, but that's just an insufficient standard by which to judge transparency reform.

Comprehensive indexes and audits of agency data force governments to make publicly accountable decisions about what is public and what isn't. Lists of government data shouldn't just include already public offerings, either. (New York's new open government law makes just this move, requiring lists of "public" data, allowing exceptions for anything that might be withheld for any reason, and ignoring all the information that should probably be public, but isn't.) If we can't see the decisions governments are making about what to release, then we can't change them.  FOI laws provide a basic instrument, but a broad mandate that places the data management burden on agency officials could systematically open far more information than ad hoc requests are ever likely to.

If governments can build data portals, hold competitions, and spend huge sums of money on complex data systems, they should be able to build public lists of those systems, along with a description of what is public and what isn't. Archivists have done this for a century for old records, and it's time for a similar amount of rigor to be applied to transparency policy decisions.

The missing open data policy is the one that says list your data, and say which of it is open, and why.

Last year, we wrote about the fatal Upper Big Branch mine explosion that killed 29 people in West Virginia. You can read all about it here, here and here.

Although the story of the Massey Energy-owned mine is controversial for many reasons, we were most concerned with the issues that related to the mine violations data collected and processed by the Mine Safety and Health Administration (MSHA), a government agency. (See what we mean in the video at the end of this post.)

Unfortunately, recent events reveal that there are greater transparency issues afoot: After West Virginian Sen. Jay Rockefeller called for Senate hearings on the Upper Big Branch Mine disaster, a subsequent investigation revealed that Massey Energy kept two separate records - a more accurate one for themselves and a cleaned up version for the federal regulators.

In a report by the Washington Post, Kevin Stricklin, a MSHA official said;

Managers were aware that chronic hazardous conditions were not recorded. What they’re required to do is list all the hazards in the official book. This is the book that not only MSHA looks at ... but it should be the book that miners and other people who are going into the mine should look at so they would be aware of any conditions in the mine before they go in.

Massey sold the Upper Big Branch mine to Alpha Natural Resources in June 2011. Luckily, the new owners are much more supportive of the ongoing investigations.

Though it’s still early in the process, the results of this investigation reaffirm our call to make public information more available and more searchable, so that we can hold our government more accountable. Indeed, the previous findings by the MSHA indicated that the government officials were quick to blame the blast on natural factors that were triggered by methane gas and coal dust. Now, the federal regulators realize that the mine owners sent in fake information that did not represent what was actually going on at the mine.

The Upper Big Branch Mine disaster is a clear example of how transparency is a two-way street: This incident could perhaps have been avoided if both the regulators and the mine owners did their part in providing the kind of information that would have saved the miners lives.

The open government movement (like most of the online world) is obsessed with “unconferences” -- meet-ups, of sorts, where the participants determine the content of and lead sessions around a pre-determined theme. When done right, it can be a powerful tool for building community.

Sunlight held its first unconference, TransparencyCamp, three years ago in an effort to get the diverse groups of people thinking about and working for government transparency together. From the conversations and problem-solving that took place there, we’ve seen the emergence of some incredible initiatives - take, for example, CityCamp.

This year, we want to go further. We want to focus on government transparency not just on Capitol Hill, but where you live. So, we need your help.

Please take a minute to fill out this survey and let us know what you want to get out of TransparencyCamp.

Never been to a TransparencyCamp or even an unconference before? Not a problem. We’re still interested in knowing what open government issues interest you, what you would want to get out of this sort of experience and how we can improve on the experiences you’ve had at similar events in the past.

TransparencyCamp 2011 will be open to people from across the country. We’re relying on your input to make it the best it can be.

http://transparencycamp.org/survey

Thanks for your help.

Today, our guest post is written by Joshua Gay, a programmer, activist, and community organizer whose interests revolve around technology, government, education, and computer user freedom.

My personal interest in the Open Atrium project came about this past fall when I began volunteering to help with the Public Equals Online Wiki. The so-called "PEO" Wiki has a lot of potential for being a good place to coordinate and collaborate on state and national transparency initiatives and projects. However, the software it is built-upon, MediaWiki, needs to be highly customised in order to make it a compelling platform for a community to start using. In my efforts to customize and improve the wiki, I have been using the features and design of Open Atrium as a sort of roadmap for improving the wiki in hopes that I can make it a more useful, powerful, and compelling tool for the transparency community.

The Open Atrium project describes itself as a "part intranet, part do-it-yourself project with a kick of open source hotness," and it certainly is one of the hottest Drupal-based projects out there. Its feature list is impressive, and for many organisations or web-based communities, I could imagine it becoming the primary tool for both project management and development. Here is a quick snapshot of it's six biggest features:

Case Tracker - Open Atrium is designed around the principle of users and groups. Every group on the system can create an unlimited number of projects within the Case Tracker, and within each project you can create to-do items. Each item can be organized and prioritized according to categories or milestones, assigned to group members, and discussions and progress notifications on to-do items can be made through a nested commenting system.

Calendar - Although not feature rich as Google calendar, Open Atrium's calendar does present events in a similar, colorful fashion, supports single or multiday features, and syncs with calendars that support iCal.

Blog - This blog contains all of the basic features you would expect with nested commenting, file attachments, and granular notification system. But, what I think makes this blogging system unique is that it is integrated into the system, and therefore, blog posts can be used as a way to discuss projects and share ideas with other members of your group and community as well as with the outside world.

Shoutbox - This Twitter-like update system is a great way to share quick updates with your group members. What I like best about the Shoutbox is that it integrates a social element into the rest of the workflow.

Documents - This is a simple, but nice collaborative document editor that supports: attachments, a revision system with a nice way to compare different versions, and a nice built print function that allows you to export and share the final product.

Dashboard - The Dashboard is where the entire system comes together and gives you a snapshot of all the activity happening across your groups. It is designed around "widgets" (like iGoogle), where users can add, remove, or arrange the widgets on the dashboard however they like. And, of course, it includes a Twitter-feed widget.

One exciting aspect about the design of Open Atrium is that its developers have designed it around the principle of features being designed like "plug-ins." Hopefully, as adoption grows, we will also a growing list of optional features that you can add to your own custom instance of Open Atrium.

I believe that Open Atrium is a powerful tool for transparency, not only for its potential use by government agencies (which would be amazing -- imagine a legislative feature!), but also an important tool for the transparency movement.

This morning a number of organizations -- POGO, OMB Watch, CREW, National Security Archive, the Center for Democracy and Technology  and the Open The Government coalition-- and Sunlight sent a letter to Vivek Kundra, Federal CIO, about improvements needed to the release of High Value Datasets on Data.gov. Here are the core recommendations included. Please tell us what you think in the comments below.

As advocates for government openness, we support the Administration’s efforts to provide the public with access to information through Data.gov. We are eager to work with you to ensure the success of Data.gov and, in that spirit, write to raise our concerns with the datasets submitted by agencies to fulfill their requirement under the Open Government Directive to post three high value datasets by January 22, and to offer constructive suggestions for improving their usefulness.

As an overall recommendation, we urge you to add public representatives to the Open Government Initiative interagency working committee and ask the committee to address the problems and recommendations identified below.

Release Format and Usability by the Public

We understand one of the primary purposes of Data.gov is to enable the technology community and transparency advocates to most effectively use the data to make a direct impact on the daily lives of the American people. The format of the data plays a key role in its usability; many within the community of advocates who re-use and repackage government data would prefer data in CSV format, rather than the XML format in which many of the posted databases are provided. Accordingly, we recommend that you strike an appropriate balance between formats (such as XML) that serve the coding community and web-based presentations by agencies that can be used and understood by the general public.

In addition, some of the currently posted files are quite large, ranging upward to several hundred megabytes. Their large size undermines their usefulness for most people or organizations. The large number of currently posted datasets also makes it difficult to find a particular database of interest. We therefore recommend that if a Data.gov dataset is available from an agency through a web-based interface, Data.gov link to that interface on the dataset's Data.gov landing page. For a consumer looking for information on a car seat, for example, it would be far easier to search the Department of Transportation's online database rather than scrolling through screen after screen of raw data in XML format. Additionally, as agencies continue to post datasets to Data.gov, efforts should be made to identify those of greatest public interest that lack such interfaces and develop web interfaces that allow the data to be explored online.

Further, while we agree there is value in aggregating government data in a single site, it is questionable how much the collocation of the currently posted information on Data.gov actually benefits the public. The site is not searchable by topic and does not provide any way to bring together data from different sources on similar topics.

As an enhancement to the organization of the site, we recommend that you use tagging or metadata to enable the public to bring together information on a topic. The thesaurus that USA.gov uses provides a useful example of the needed vocabulary.

Value of Data

The release of the datasets also has prompted discussions about the value and the quality of the released data, and the additional value provided by access to existing data in a new format. We believe repackaging old information is of marginal value, yet that is what many agencies have done with their recent postings on Data.gov. According to the Sunlight Foundation, of 58 datasets posted by major agencies, only 16 were previously unavailable in some format online. This leaves the impression that agencies posted easily available data, the proverbial low-hanging fruit, rather than seriously considering which of their datasets truly are of high value. While these initial postings can be considered a test run, more attention needs to be directed toward ensuring the overall quality and usefulness of the data.

In addition, sustained attention should be paid to the possibility of making some of the datasets available as feeds that are constantly up to date, rather than as static datasets that are pulled down and then reposted on an occasional basis. We recommend that agencies be required to explain why the data is high value by having them designate which of the “high value criteria” the data meets: information that can be used to increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation. Similarly, we recommend requiring agencies to indicate whether a high value dataset was previously unavailable, available only with a FOIA request, available only for purchase, or available, but in a less user-friendly format. Going forward, this will make it much easier to track how agencies are complying with the other requirements of the Open Government Directive. While we appreciate the value of data that furthers the mission of an agency, we believe it is equally important to make available to the public data that holds an agency accountable for its policy and spending decisions. We hope to see more datasets of this type available in the near future.

Quality

As is to be expected in efforts of this type, there were a number of glitches--datasets that could not be downloaded or, once downloaded, could not be opened (the Central Contractor Registration FOIA extract from the General Services Administration seems to have caused several users problems). Additionally, some datasets were incomplete (the Hazard Grant Mitigation Program data released by FEMA is missing 23 years of data between 1966 and 1989). Even more troubling, some did not have header rows, and for those that did, their Data.gov pages did not always link to code sheets explaining what those header rows meant. Without this information, the data cannot be used.

We therefore urge the implementation of a responsive feedback mechanism that allows the public to alert an agency that a specific dataset is not working, lacks information, or is missing explanatory material and provides a response to the concerns within a specified time. One way to address this may be to include an agency contact with the ability to resolve any database problems or provide information about the database. The interagency working group could sample the quality of these agency-specific dialogues to ensure that they are having an impact and to develop recommendations on best practices to improve the responsiveness. Additionally, we strongly recommend that all datasets on Data.gov be directly associated with their code sheets.

Finally, we are concerned with the current lack of public notice when data is removed from the site. We respectfully urge you to note all raw tools and data that are removed from Data.gov, and to provide an explanation for their removal.

Many of the concerns outlined above apply across all or many of the agencies’ datasets. Accordingly, we think that standards for handling these types of problems can easily be addressed through the interagency working group and then disseminated amongst the agencies.

David Moore at Open Congress has an excellent post up explaining how the current life of a bill in Congress is riddled with disclosure holes. I can't do more than say, go read David's post. Here's some choice graphs:

The reason is that the “Baucus Bill” is only a “mark”, not yet an official Senate bill, which means (to summarize reductively) that the digital text that constitutes the .pdf does not make its way off internal government web servers to the official website of the Library of Congress, THOMAS — and in turn, does not make its way to government transparency web resources such as GovTrack and OpenCongress. Before that happens, this mark of the health care bill needs to be reconciled with other Senate committee versions of the same, which will then be put forward for consideration to the U.S. Senate as a whole. Health care reform is leading news coverage & blog analysis of American politics right now, this is a major document in the mix, and there’s not a widely-recognized, user-friendly resource for online examination by the public at large. You should have better access to this info! You should have — at your fingertips — immediate, unrestricted digital access to the full text of any piece of legislation the very moment it’s released publicly by Congress.

...

The current Congressional process for publishing data is, to borrow a phrase from the Free Software Foundation, Defective By Design. As we see in many proprietary, top-down systems affecting the public interest, it’s insistently closed-off. Congress’ processes for distributing legislative info is fundamentally broken — it could and should relatively easily be fixed, starting now. Whether or not you support the Baucus markup or the House version of the health care reform bill, we hope you agree that the public has a right to read this important iteration & political volley in the process.

Two nights ago, in his The Word segment, Steven Colbert actually used his show to do some investigative work into the money-in-politics connections that might have motivated to turn Rep. Luis Gutierrez' position on pay-day loans from oppose to support slight (read: non-existent) restrictions. Watch it:

The Colbert Report	Mon - Thurs 11:30pm / 10:30c
The Word - Have Your Cake And Eat It, Too
colbertnation.com
Colbert Report Full Episodes Political Humor NASA Name Contest

Of course, I can't help but remind readers that the Center for Responsive Politics recently opened up their data--20 years worth--to be mashed, mixed, and visualized. Already we are seeing visualization pop up. Check out these from the University of Michigan.

Also, Sunlight's Chief Evangelist Greg Elin penned a guest column over at ProgrammableWeb about the release of this mountain of data. It's well worth the read.