The G8 countries today released a new declaration, and with it an Open Data Charter policy paper, which together constitute a significant high-level commitment to open data and transparency.

Sunlight has been close to the ideas, movement, and conversations that have helped lead to this announcement.  We're thrilled to see such a visible, detailed statement from the G8, moving what has long been a national level issue, and more recently a multi-stakeholder issue, to now an idea jointly promoted and celebrated by some of the world's most powerful governments on their own terms.

This statement provides helpful language in its commitments, reiterating the open data benefits canon, and imposing a new reporting processes and avenues for collaboration, both intergovernmental and across sectors. It will likely help prioritize open data initiatives in other policy venues, and empower internal reform champions within G8 governments (and beyond) to pursue open data.

These results are predictable, in part, because they're the kinds of impact we've seen from the succession of highly visible US open data policies over the last 5 or so years, culminating recently in the newest Open Data Policy from the administration.

The experience of those successive policies also suggest some likely challenges that the G8's Open Data Charter will soon face, for example: the limits of a head of state's voluntary commitment in affecting civil servants, the difficulty in articulating "high value" data (in even a single country context), the challenge in operationalizing the "default" of openness, the gap between governments' open data commitments and their transparency performance on other issues (especially those that affect political power), or the inherent difficulty in evaluating openness, when only the disclosed is accessible.

Each of those issues, and many others, will now play out across the G8 countries, and we'll all be better off for it. Open data is a concept that should touch any number of other issue domains within government, and a high-level commitment to openness is among the best ways of taking the spirit of openness that underpins our ideas about democracy, and applying it to all the places where we haven't yet managed to apply it.

It's a testament to the optimism and potential we see in new technology that our some of our broadest hopes for shared knowledge are now being expressed in terms of Open Data, at the highest levels of governments around the world. Let's work to ensure that our expectations, vision, and judgment are proportional to the challenge.

Sunlight is excited today to announce, with a few of our friends and allies, the Global Open Data Initiative, dedicated to helping lead on open data issues in a global context.

For much of the last year, we have been grappling with the opportunities and challenges created by governments around the world as they commit to new open data policies and initiatives. Indeed, the most frequent commitment governments have made in their action plans for the Open Government Partnership are about open data. And while this enthusiasm is clearly useful, these commitments are being made with often unclear expectations, with both governments and civil society groups searching for guidance and standards to judge governments and their open data performance.

We’ve decided to see this as an opportunity, since an abundance of political enthusiasm should be a huge opportunity for civil society groups to take this idea of “open data” and make it as meaningful as possible.

At Sunlight, in our domestic experience, this has meant approaching open data as a means to create accountability, deepen civic life, and empower new forms of activism, analysis, and reporting. We’ve nurtured a conception of open data that is grounded in technologists’ work and that is savvy to political power.

We’re thrilled to be bringing our vision for open data into a partnership with groups from around the world who have extensive experience with different aspects of open data. Working together, Sunlight and our partners -- Fundar, Open Knowledge Foundation, World Wide Web Foundation, and Open Institute -- will use the platform of the Global Open Data Initiative to:

• Provide a leading vision for how governments approach open data
• Increase awareness of open data and related issues.
• Support the development of the global open data community.
• Gather, expand, and amplify the evidence base for open data.

Open data -- and what it means -- cannot be owned by any one organization or group. In a sense, it’s just a new approach to creating access to public knowledge. If we’ve done our job right, the Global Open Data Initiative will serve as a platform to advance this end -- to highlight valuable work, create new ideas, and help amplify others’ voices on open data around the world. We hope you’ll consider joining our discussion list and look forward to working together.

We’re also excited to begin working with other similar initiatives from other related disciplines -- such as the International Aid Transparency Initiative, Extractive Industries Transparency Initiative, Natural Resources Charter, Construction Sector Transparency Initiative, International Budget Partnership, and OpeningParliament.org -- because open data intersects with their work and because we’ve taken significant cues from their design in approaching our own.

Finally, we are thrilled to acknowledge and announce the financial support of Omidyar Network and the Open Society Foundations, whose guidance and advice have been invaluable in designing this new initiative.

More about the Global Open Data Initiative on our website: http://globalopendatainitiative.org/.

While Sunlight has generally not been a target of Evgeny Morozov's criticisms, I was amused a few days ago to discover that I'm quoted in his new book, To Save Everything Click Here (page 117). Morozov suggests that Sunlight is a "bastion of technosolutionism," quoting me from this 2010 Economist article: "There is a cultural change in what people expect from government, fuelled by the experience of shopping on the internet and having real-time access to financial information."

Technology is at the heart of Sunlight's approach to our mission, and we do see citizens' technology-fueled expectations for information access as an overwhelmingly positive force in pushing for a more transparent, accountable government. (Indeed, other versions of the quote Morozov cites can be found throughout our work. I suspect that Morozov has overlooked the valuable work of our peers as well, whose ideas and work often inspire our initiatives.)

But accusing Sunlight of being a "bastion of technosolutionism" misses the mark enormously. I was initially torn over whether to write about the skepticism, nuance, and substance that help fuel Sunlight's work, but then I came across another passage from Morozov that provides a perfect opportunity:

How do we ensure accountability? Let’s forget about databases for a moment and think about power. How do we make the government feel the heat of public attention? Perhaps by forcing it to make targeted disclosures of particularly sensitive data sets. Perhaps by strengthening the FOIA laws, or at least making sure that government agencies comply with existing provisions. Or perhaps by funding intermediaries that can build narratives around data—much of the released data is so complex that few amateurs have the processing power and expertise to read and make sense of it in their basements. This might be very useful for boosting accountability but useless for boosting innovation; likewise, you can think of many data releases that would be great for innovation and do nothing for accountability.

In this passage, Morozov outlines what he sees as a good faith approach to creating government accountability through open data. This passage reads like an intro paragraph to a Sunlight Foundation strategy document or activity report. Let's try it again with links:

How do we ensure accountability? Let’s forget about databases for a moment and think about power. How do we make the government feel the heat of public attention? Perhaps by forcing it to make targeted disclosures of particularly sensitive data sets. Perhaps by strengthening the FOIA laws, or at least making sure that government agencies comply with existing provisions. Or perhaps by funding intermediaries that can build narratives around data—much of the released data is so complex that few amateurs have the processing power and expertise to read and make sense of it in their basements. This might be very useful for boosting accountability but useless for boosting innovation; likewise, you can think of many data releases that would be great for innovation and do nothing for accountability.

I could have added links to each letter instead of each word. Sunlight's work is varied and complex, and definitely can't be adequately summarized as "solutionist" -- Sunlight lives at the intersection of journalism, advocacy, technology, and political power.

My screen shows 579 pages of blog posts on just the main Sunlight blog, covering our activities since our founding in 2006. If you read all of them, no doubt you could extract some sentiments that you could call "solutionist." You'd also find skepticism, optimism, public dialog, exuberance, curiosity, and all the various signs of people in an organization engaged publicly in the complicated work of government transparency.

We think about risks like "solutionism" (I referred recently to the "triumphalist" view of legislative versioning), just as we think about the risk of relying too strongly on an inside (or outside) advocacy strategy, or the risks of advocating for transparency in the wrong places. (You'd be unlikely to conclude that Sunlight endorses a "monitorial democracy," though, if you were familiar with the depth of our writing and thinking on the topic.)

Sunlight's conception of open data is not agnostic to political power, refuses to be reduced to splashy data portals, and serves specific needs to help create valuable journalism and public dialog and a more substantive politics.

The Center for Public Integrity has an important story today on Organizing for Action's money in politics problem:

Organizing for Action, launched by former Obama campaign officials earlier this year, confirms it will not publicly release donors' employer and occupation data despite collecting it through its online donation form.

Lisa Rosenberg made a similar observation recently, in adding incomplete disclosure to the list of problems with the President's dark money group. To reiterate: OFA allows the President to accept unlimited donations (in the name of grassroots organizing), to build political power for the President, while avoiding the reach of campaign finance laws.

Here is President Obama in 2010 (video, transcript), talking about the DISCLOSE Act:

These shadow groups are already forming and building war chests of tens of millions of dollars to influence the fall elections. Now, imagine the power this will give special interests over politicians.  Corporate lobbyists will be able to tell members of Congress if they don’t vote the right way, they will face an onslaught of negative ads in their next campaign.  And all too often, no one will actually know who’s really behind those ads.

Now it's 2013, Obama has his own "shadow group." We already knew OFA's "voluntary disclosure" was ultimately unreliable. (It's voluntary, and there's a reason campaign finance laws aren't designed to be self-enforcing. On top of that, even if they know the donor's identity, donors are still quite capable of laundering their donations.)

And now we get official confirmation that even though OFA collects important information about donors, and they plan not to release that information publicly.  Why would Obama's group want to know this information, while at the same time keeping it from the public? Why does the President seem to have no plan for OFA's structure, apparently making it up as they go?

The President who was so concerned about the corrupting effects of unlimited contributions is surprisingly unconcerned about the effects of unlimited contributions on himself, and his c4 reflects it. OFA's ability to broker access to the President and tap into large donors is only being held back by the threat of public blowback, and not at all by any Presidential sense of accountability. If OFA's structure were motivated by accountability, we'd see a coherent policy about campaign finance disclosure, empowering public oversight of his group's finances and donors. Instead, we see conflicting messages about what kind of access a $50K donor can expect, and a disclosure policy that exists only in proportion to public outrage about Obama's dark money.

President Obama should be leading the way in creating accountability in our politics. Instead he's leading politics to a place with little accountability.

There's a certain conventional wisdom that President Obama wants stronger campaign finance laws, and to protect our democracy from the corrupting effects of money in politics.

It's a story that you should no longer believe.

The arc of the Obama presidency may be long, but so far, it has bent away from transparency for influence and campaign finance, and toward big funders.

There's the obvious examples, like promising (and failing) to put healthcare negotiations on C-SPAN, only to negotiate a secret agreement with a segment of the industry the reform effort sought to regulate.

But there's a longer pattern here too. Obama came to power as the outsider who would return merit to public policy, and raise up regular citizens' voices in the process, at the expense of the moneyed interests whose power had displaced regular people. (My career was kick-started, in part, by helping to inspire an amendment to a 2006 Senate bill sponsored by Obama and Salazar, an idea emerging from a series of blog posts on Daily Kos.)

Obama was the standard bearer for post-Citizens United reform, sparring with Republican Senate opposition to ultimately fall a vote short of passing the DISCLOSE Act. 2010 was the high point for Obama's campaign finance rhetoric, where weekly speeches and Rose Garden addresses sought to affirm the dangers of dark money, and the need to understand whose money is buying Washington.

How far we've come.

Since then, Obama embraced superPACs and c4s, the vehicles of newly deregulated influence-buying, suggesting that they were a necessary evil, and suggesting that the only alternative would be unilateral disarmament. A reluctant participant in a rigged game. The rationale didn't ring true even then -- the weak disclosure for these groups was justified by appeals to the same broken laws Obama spent 2010 railing against.

Then came the inauguration.

Obama reversed his policy of limiting donors for the inauguration celebration, and failed to post donation amounts online. (George W Bush had amounts and ranges online for his inauguration donors). He sold access to the inauguration, and to the Presidency, to corporate donors, wheedling their way into the privileged positions that move their policy agendas forward. Obama did this for a series of parties, with no public interest justification whatsoever, and didn't disclose donor amounts. The transparency President couldn't publish donor amounts on the internet -- of corporate donors. Private citizens can make hundreds of GIFs of a comical Rubio water swilling incident within 3 minutes, but the President of the United States can't post corporate donation amounts online.

Transparency President no more.

And now we've got the new c4. It's hard to overstate how bad this new policy is for campaign finance. Today's LA Times (via Democracy 21) explains that Obama's new c4 has been set up to sell direct access to the President, for huge sums of cash, which will be disclosed online, quarterly, without specific dollar figures. The President who told us that secret money in politics undermines democracy has now created a huge funnel for donations, with accompanying disclosure that would have been considered cutting edge in 1992. Quarterly disclosure in ranges is the kind of disclosure you create when you don't want to be seen. This should be a policy innovation Obama is remembered for -- a return to soft money and unlimited donations outside the confines of campaign finance law, with instant access to the White House for the most well-heeled donors, all, incredibly, in the name of empowering the grassroots. It's more egregious, direct, (and potentially corrupting) than the similar efforts of recent Presidents who came before him, an evolutionary step forward for money in politics that is more legal, more normalized, and more powerful than it was before.

Maybe Obama has stopped talking about Citizens United because he's learned to use it to his advantage. Maybe his campaign finance rhetoric was fake all along, donning the visage of a reformer. It doesn't matter.

Obama is taking on money in politics by getting more money into his politics.

Obama has done valuable things for transparency. There are innovations that create value for everyone, and many good people in the White House have created valuable things that continue. But we should be clear about Obama's position on transparency when it comes to his political power.

It's time to stop worrying about how Obama can help fix campaign finance, and instead worry about how we fix what he's created.

In years past, we have celebrated or analyzed the transparency, ethics, and money-in-politics themes from State of the Union addresses.

This year, there's not much to talk about.

Aside from saying that the White House will ensure anti-terrorism "efforts are even more transparent to the American people and to the world" (whatever that means), transparency didn't come up at all.

Obama says that we have to "keep the people's government open," but he means avoid a shutdown, not remain transparent.

This is a far cry from recent years -- here are excerpts from 2012, 2011, and 2010.

It's not a surprise for government integrity, transparency, and influence to be absent, since Obama's visionary rhetoric and action on government transparency have largely been replaced with rhetorical defenses.

Maybe it's still too awkward to talk about lobbying and dark money when the President is accepting huge corporate donations for his inauguration parties and designing his own c4.  By his own words, the post-Citizens United Obama repeatedly assured us that secret money in politics would undermine integrity. We're still wondering whether his ongoing silence on transparency and money in politics is evidence that he was right.

As we started thinking about how to approach cities across the US, we had to think about where to focus our effort and attention. Should we focus on a few key, well-known cities who set the strongest examples of developing transparency and open data reforms? Should we narrow down the list to only the biggest, say, 10 cities, or break them into categories by size?

In confronting these questions, we found important lessons about the population sizes of American cities that we’ll use to shape our overall approach to our municipal work. It turns out that most U.S. cities are far smaller than you might think. More than 80 percent of U.S. cities have fewer than 10,000 people. The scope and kinds of data housed in these cities might be vastly different from the data in a metropolis like Los Angeles, with its more than 3 million people in the city limits alone.

We also broke down the data to look at the distribution of cities with fewer than 500 people.

The country’s 25 most populous cities are home to about 10% of the U.S. population. Perhaps more surprisingly, another 10% of the population lives in cities with populations under 10,000. So what does that mean? That small cities -- and towns -- are important, and that the transparency and accountability policies of small cities matter enormously.

Small and medium-size cities make up a large portion of the municipal governments in the United States. (Again, 80% of U.S. cities have populations of fewer than 10,000 people.) The fact that there are so many small and medium sized cities suggests something about our country’s growth and development. Just as cities have been drivers of cultural and industrial development, they are also a window into the country’s diversity and history. Engaging with the richness of these small and medium-sized places means engaging with the richness and complexity of our whole country. It gives us greater perspective on the cross-section of people and governments entangled in the questions of local accountability that we’re trying to answer and is a good reminder of why we have taken on this work in the first place.

Laurenellen McCann, Cat Robinson, Zander Furnas, and Alisha Green also contributed to this post. 

We used 2010 US Census data to explore the wide range of populations among incorporated places. We use “cities” here in a colloquial sense, referring to the range of “incorporated places” tracked by the Census, which includes everything from cities and towns to villages and boroughs. The data for the pie chart is from the 2010 US Census Incorporated Places by Population Size file. Data for the histograms is from the National Places Gazetteer Files.

Because of some the work we've done before on last minute negotiations and divided government, Sunlight prepared the following graphic that visualizes the recent history of US House votes on the debt ceiling, based on public voting records and a CRS report. The bars indicate the "yea" votes.

We'll have more commentary forthcoming, but here are a few initial thoughts on what this graphic makes clear:

1. Opposition to raising the debt ceiling is often partisan, with opposition coming from either party, based on who is in the White House. Many House Republicans have voted for raising the ceiling, just as President Obama voted against it when he was a Senator.
2. Divided government has necessitated support from both parties to raise the limit.
3. There is a significant untold story about the Gephardt Rule, a House Rule which enabled the limit to be raised with little public record. The role this rule played in setting up the current showdowns has been insufficiently examined.
4. Good access to congressional data and reports enables this kind of analysis; it could be improved.
5. Each of these votes was a predictable consequence of budgets that were passed before them, demonstrating another facet of political hypocrisy.

Daniel Schuman, Zander Furnas, Caitlin Weber and Matt Rumsey contributed to this post.

As we expected, the culmination of the "fiscal cliff" negotiations was a rush to the finish line, in which policies decided by a few men in a room were passed through the Congress without amendment. The last few days of congressional deliberations were so bad they make the supercommittee look participatory, or, as Tom Fitton of Judicial Watch observed, they make "the passage of Obamacare seem like the apex of republican democracy."

Some observations on the mess:

The Unread Bill: Despite Boehner's many promises to the contrary, the bill was not online for 72 hours. This isn't a surprise, since the House wasn't really a party to the final negotiations, and relegated itself to the binary position of pass or veto usually reserved for the President. The House voted blind, with about 12 hours of access to the bill, starting New Year's morning, before floor debate. The only bright spot here is that the House Rules Committee helpfully posted the bill as soon as they had it. Of course, that's nowhere near adequate time to process a bill, so stories of the ridiculous corporate tax subsidies in the bill only came out as the floor debate started.

This is a side effect of not legislating. If Congress legislates, then they'll have access to legislation. If they try to maximize imaginary leverage through crisis brinksmanship, then their role gets diminished to the point that they're voting on things they haven't read.

Someone Had Access: While Congress (and the rest of us) only just found out what was in the bill, a coterie of corporate lobbyists managed to get their profit-boosting tax expenditures included. It's hard to imagine how NASCAR and Hollywood had stronger negotiating positions than the House of Representatives, but in the end, they did.

Faith, Lost: One fake disclosure and staged leak after another punctuated the entirety of the negotiations, from CPI to brave phone calls to congressional curse words. In the absence of substantive proposals, the public was cast into the world of fake narratives and reassurances. Our party leaders provided no check whatsoever against manipulative PR negotiating tactics.

What Deal: We cautioned in November that we are unlikely to know the outlines of the entire deal, since concessions could be made outside the context of legislation. (This is not a hypothetical concern).  This seems less likely to have happened in yesterday's deal -- the process was so convoluted that secret deal terms would probably have been impossible to agree to. The fact remains, though, that when the negotiating table is utterly unseen, we can't know what's put on it in our names.

Groundhog Day: We'll be at it again in a few months, and it's likely to get worse.  Each iteration of these negotiations has produced less public information -- heightening fearmongering, and uncertainty, while further empowering those with privileged access.  It's probably going to get worse before it gets better.  Our recommendation still stands -- Refuse to cynically accept secret negotiations as the status quo for self-government.  

Every day in Washington, DC, thousands of government officials are at work writing, revising, and publishing regulations affecting everything from food safety, to personal finances, to the environment and beyond.

But as our Reporting team reports today, with the help of data crunched by Sunlight Labs, the rulemaking process, though ostensibly a public one that anyone can participate in, is largely a rarified one. But it doesn’t have to be that way.

This year 89% percent of the regulations that drew at least one comment received 100 or fewer, according to an analysis of information available on regulations.gov -- the government's one-stop hub for information about the regulatory process -- that we made using Docket Wrench, a new tool in development by our Labs team. Only 102 regulatory proposals drew 1,000 comments or more.

In contrast, the regulations that drew the most participation were largely concerned with controversial matters that are patrolled by groups with well organized constituencies eager to act when prompted by an action alert. For example, the regulation that received the most comments—more than 63,000—concerned contraceptive benefits under the Affordable Care Act. Much of the action appeared to be organized by Catholic groups opposed on religious grounds.

We are hoping that Sunlight’s Docket Wrench, which is based on data collected by regulations.gov, will draw more people in to the process. Our Scout tool also offers alerts on regulations as published in the Federal Register.

However, our tools can only be as good as the data underlying them. While a long list of federal agencies participate in regulations.gov, technically independent agencies are not required to do so. These include such high profile agencies as the Securities and Exchange Commission, the Federal Communications Commission, and the Federal Elections Commission. While these agencies individually may post the comments they receive on pending regulations, very few have an “API” that allows developers to pull the data into a new interface as Labs does with Docket Wrench. Without the participation of these agencies in regulations.gov, our new Docket Wrench tool will be limited.

In addition, the data collected at regulations.gov is often imperfect. There’s little standardization of fields, such as “employer,” that would allow a user to easily search for a particular entity. When our reporting group sought to find out, before the election, where the U.S. Chamber of Commerce was putting its regulatory efforts, we found ourselves limited because the data would not easily allow a thorough search on the name of the organization. Ideally each organization filing a comment should use unique corporate identifiers that would help make such searches comprehensive.

It should be mandatory that all of the federal agencies -- including the independent ones -- should participate in regulations.gov. Of course the agencies should receive the funding necessary to make this happen. If we can improve the data, that’s one step closer to improving participation.