Residents of Minnesota now have a new way to keep track of what’s happening in their state with the launch of OpenGovernment Minnesota. The “land of 10,000 lakes” is the latest state added to OpenGovernment, a joint project of the Sunlight Foundation and the Participatory Politics Foundation, along with support from the Minnesota Historical Society.*

Visit MN.opengovernment.org to get the real story behind what's happening in government across the state via official government information, local news coverage, blog posts and social media alerts.

Writes David Moore on the OpenGovernment blog:

Now folks in Minnesota can track with ease everything their state legislature does — all the bills that are proposed, votes that are taken, money that was raised, and more. We’ve timed the launch of this, the sixth U.S. state on OpenGovernment, to coincide with the Netroots Nation conference ongoing this weekend in Minneapolis / St. Paul. We’re pleased to share this new public resource for accountability in government and citizen watchdogging with all the political bloggers & issue-based activists there.

The Sunlight Labs Open States project developed the legislative backend for OpenGovernment. Supported in part by the work of volunteers, the Open States project’s goal is to collect and scrape legislative data from all 50 state legislatures and make it available online in a unified, developer-friendly format.

* Update: Additional support from the Library of Congress National Digital Information Infrastructure and Preservation Program.

 

Today the Sunlight Foundation is proud to unveil Sarah's Inbox, our attempt to make Sarah Palin's recently released email records easier to use with a searchable function and an interface similar to Gmail. It builds on Elena's Inbox, our wildly popular project launched almost exactly one year ago that took the email data of Supreme Court justice Elena Kagan released by the Clinton Library and made it more accessible online.

Sarah's Inbox allows users to view the more than 14,000 emails from Sarah Palin's tenure as Governor of Alaska with familiar sorting functions. You can go page by page starting from the most recent emails or, most importantly, search. To help direct folks to interesting items, try some of our sample searches, star emails for later viewing or view the most starred emails by all users.

The project started after we were again approached by folks on Twitter and the Sunlight Labs list (join!) to take this ugly data and add the Sunlight secret sauce to make it user friendly. Initially we were cautious because the cast of characters who directly obtained the data included the likes of the New York Times, ProPublica, Mother Jones and MSNBC.com. We spoke with ProPublica and they encouraged us to take a stab at fashioning our own tool, so we borrowed their data and went to work. Sarah's Inbox would not be possible if not for the great people at Crivella West to gather, lift, scan and pay for all this data.

Like Elena's Inbox, Sarah's Inbox faced staggering issues of data quality because government officials continue to release digital files as hideous printouts requiring a laborious and error-ridden optical character recognition (OCR) pass over. You will notice that many of the emails are garbled, incomplete or contain odd characters - please keep in mind that we did the best with what we had and are not responsible for the content. Due to the programmatic nature of the tools used to build this site, we recommend checking any research effort against the source files.

Disclaimers aside, please enjoy Sarah's Inbox and tweet interesting items you find with #sarahsinbox.

This week marks the fifth anniversary of the Sunlight Foundation. It is exciting to reflect on how far we've come, the great people who helped us along the way and where we plan to go. With your help, we've grown from a small organization with big ideas to a connected community whose call for greater government openness and transparency is heard throughout the country.

We began with the nonpartisan goal of using the revolutionary power of the Internet and new technology to open government information. When we started, this modern interpretation of transparency was almost a completely foreign idea in Washington - a place where corrupt lobbyist Jack Abramoff dominated the headlines and sifting through reams of paper in order to get at the truth of what was going on was the status quo. While ordinary citizens were embracing new media tools and websites that gave them a readily available stream of information at their fingertips, government showed little interest in keeping up with the times.

Right out of the gate, we took on these age-old issues with a fresh arsenal of online tools and empowered citizens to engage in new forms of direct oversight. We believed then, as we still do, that none of us are as smart as all of us and that we have a stronger democracy when open information gives people the ammunition they need so they can speak truth to power. Sunlight developed all kinds of new tools and websites to achieve these goals and get the public involved in the political process.

We encouraged lawmakers to post their schedules online and launched the Open House Project to engage policy experts, citizens and lawmakers in a conversation on all the ways the House of Representatives could update how it shares information with the public. We initiated and funded dozens of projects to create online databases of government information. Everything from earmarks to congressional fundraiser invitations to foreign lobbying disclosures to House expenditures. We created mobile applications to put Congress into the hands of the people and fostered a community of thousands of 'civic hackers' to build better tools. We updated legislative rules and collaboratively wrote new policy to open government to the Internet age. We've trained thousands of journalists and citizens in using data and in using the web to watchdog Washington. We modeled government websites to show what is possible and followed the money, lobbying and the influence industry with ongoing reporting projects.

Through it all we are most inspired and proud of the people who take action and participate in the process to improve our democracy. Thank you to the countless people and organizations who have worked with us, used our tools, dug deep into our websites through our first five years. The Sunlight Foundation will continue to work with you explore how to enhance our democracy and citizen engagement with our public officials using online tools. Sure, there's a lot more to be done. As a wise person once said, if this was easy, it would have been done already. And we promise you - the best is yet to come!

Please continue to support our work to keep the light shining on government.

Last night the Sunlight Foundation's award-winning Sunlight Live platform covered the State of the Union with context and fact-checking from Sunlight's Reporting Group and teams from the Huffington Post, National Journal, CQ Roll Call and the Center for Public Integrity. It was an exciting evening and we're honored so many of you chose to join our coverage for Obama's speech, the Republican response and the Tea Party followup.

More than 10,000 people tuned in and, while at times we were nearly technically overwhelmed by the response, our talented Sunlight Labs team held us together. The engaged viewers left over 1,000 comments and we published more than a third of them to be answered by the reporters or shared with other visitors. Hundreds of folks camped out on the site hours before the speech, indicating their preference to watch on our channel. As best as we can measure, 2,308 tweets and 908 shares on Facebook sent fans to Sunlight Live.

Here are some excerpts from the various news coverage our Sunlight Live project received:

Roshan Nebhrajani from Medill's School of Journalism joined Sunlight in our office and reported on the experience:

A group of 14 reporters gathered at the Sunlight Foundation, all centrally connected by one crucial link — a heavy-duty extension cord — as they typed through dinner to provide interactive coverage of the address to nearly 2,000 viewers. [...] visitors to the Sunlight Foundation’s site engaged in online conversation. One said: “This is perfect. Like sitting in the room, watching with a bunch of smart, informed people.”

GOOD Magazine initially promoted the White House live-stream online but switched to support Sunlight Live after learning the extent of our coverage:

While we indeed support the government smartly using technological advancements to spread information, in this case, we're going to direct you away from the White House's stream and toward the Sunlight Foundation's live blog. Not only will Sunlight be streaming video of the address, reporters from CQ Roll Call, the National Journal, the Center for Public Integrity, and the Huffington Post will be on hand to fact-check and offer context as the president speaks. We can almost guarantee that the information provided will be more objective and less dry than what the White House is offering. Happy viewing.

Fast Company did a roundup of all the various ways to watch the State of the Union and highlighted the collaborative and real-time reporting during Sunlight Live:

Traditionally, we’ve had to wait for the networks’ post-game shows before anyone starts to dissect the accuracy of various statements made by the president or the opposition. But last night, the Sunlight Foundation—in partnership with The Huffington Post, National Journal, CQ Roll Call, and the Center for Public Integrity—posted real-time fact-checking during the course of the addresses.

MediaBistro has an article about the new dawn of real-time fact checking that points to the work of the Sunlight Foundation and the Sunlight Live event:

Gone are the days when political junkies would have to wait for a speech to be over before talking heads could endlessly parse each word. [...] with our incredible shrinking news cycle and the rise of participatory journalism, the approach only makes sense.

It was a great team effort at Sunlight and we loved working with our partners from the Center for Public Integrity, National Journal, CQ Roll Call and the Huffington Post. Thank you to everyone who helped make this Sunlight Live event a success and we hope you join us for future coverage.

Photo by Nicko Margolies

The Sunlight Foundation is proud to announce our Checking Influence tool that gives individuals the power to see the political expenditures of the businesses you frequent. The simple bookmarklet allows users to connect personal spending habits seen on your online bank or credit card statement with the lobbying and political contributions of companies.

As we start to examine how much we spent on Black Friday or Cyber Monday, Checking Influence will let all of us see how effortlessly politics escapes Washington and settles its way into our wallets; often without us even knowing it. We created this tool to help Americans be more informed consumers and citizens. Just as some consumers check to see if their coffee is free trade or if their clothing is manufactured in sweatshops, they can now know if their purchases help fund lobbying campaigns. We’re trying to answer the question: When you buy coffee at Starbucks; refill a prescription at Walgreens; or download a song from iTunes, do you know where your money really goes?

How to Use Checking Influence

Using Checking Influence is simple and secure. First, add the Checking Influence bookmarklet to your browser’s toolbar. Next, go to any web page that shows your spending transactions, such as a banking site, your credit card statement or Mint.com. Then, just click on the Checking Influence bookmarklet, and it will find the company names on the transactions list and show you the “influence data” for the corporations it can identify -- including political campaign contributions and what lobbying the corporation conducted.

Behind the Curtain

The backbone of Checking Influence is TransparencyData.com, Sunlight’s open-source, central repository for federal lobbyist registrations, federal grant and federal and state campaign contributions. Sunlight Labs, the Foundation's in-house team that builds technology and Internet tools to make government more transparent and accountable, developed Checking Influence. The site is built upon the public Transparency Data API, whose data is provided by the Center for Responsive Politics and the National Institute for Money in State Politics.

A Note About Security

We understand that everyone is cautious about banking information online (and rightfully so!), which is why the Sunlight Foundation has taken a number of steps to ensure that Checking Influence is safe to use. Checking Influence uses the same industry-standard SSL encryption that your banking site does to keep your financial information secure and we don't save any personally identifiable information. The tool is simply searching bank statements for transactions with company names that match information from TransparencyData.com.

Please contact us with any feedback and we hope you enjoy playing around with this new tool!

by Sunlight Foundation policy intern Melanie Buck

The role technology can play in streamlining the declassification process was the topic of a Public Interest Declassification Board meeting on Thursday, Sept. 23. The PIDB is an congressionally-established advisory committee that works to facilitate public access to national security-related records. It is considering how to advise agencies on their efforts to declassify approximately 410 million pages of records by December 2013. An agenda for the meeting is available here [PDF].

The Board heard presentations on the feasibility of using an automated computer systems to streamline document review. The speakers were Jeff Jonas from IBM, Tom Lee from the Sunlight Foundation, and John Verdi from the Electronic Privacy Information Center.

Jeff Jonas outlined a hypothetical automated system that would tag documents based on key words and phrases to make predictions about whether a document should be declassified. A high level of accuracy would come from training the system on documents that have already been reviewed, combined with determining how the new document relates to the old information, in a process known as "context accumulation." He analogizes context accumulation to solving a jigsaw puzzle in this blogpost. The technology already exists, but would take some time to implement.

Tom Lee described the requirements for determining whether a document should be declassified, focusing on how a computer system could help prioritize the work queues of reviewers. For example, pages that the system determines most likely to be sensitive can be reviewed first, and if the system is determined to have made a correct judgment, the rest of the document (and potentially the document series) can be removed from the work queue. A search algorithm could be trained to return sophisticated results that would give human reviewers a clear indication of the content of a given document. Such a system would involve a static up-front cost, with additional computational costs varying on the system's operational speed.

John Verdi approached the issue from a policy perspective, explaining his evaluation of what transparency groups and the public want from a declassification process. He suggested that the preparation of unclassified summaries adds work without adding much public value and is an unnecessary burden on the declassification process. In his opinion, resources would be better spent reviewing entire documents and declassifying whenever possible. He also discussed a few transparency tools that many hope to see, such as a large, openly-accessible searchable database of declassified records.

The Board has another meeting scheduled for Nov. 9, 2010 to further investigate ways to facilitate declassification.

Declassification has been a current focus in Congress as legislators promote a cultural shift from “need to know” to “need to share.” Just last week, Congress sent the "Reducing Over-Classification Act," H.R. 553, to the President for his signature. Among other things, the legislation requires the Director of National Intelligence to establish policies and procedures to identify the classification of portions of information within an intelligence product, hopefully thereby facilitating automated review.

We summarized a July 22 discussion of the declassification of historical congressional records here.

As I noted in my speech yesterday at the Gov2.0 Summit, Gov2.0 has become a popular catchphrase in Washington today and no organization has been more excited about its potential when it comes to data transparency than the Sunlight Foundation. But now, some 20 months since President Obama made his initial commitment to technology and transparency, we have numerous concerns. One of the core examples that I used was USASpending.gov, which President Obama championed into law when he was in the Senate, along with Sen. Tom Coburn, in 2007.

USASpending was created to provide the public with information about how the federal government spends our tax dollars. It was launched nearly three years ago and has already gone through three redesigns, each one flashier than the next. The site is pretty impressive graphics-wise, but unfortunately the data provided is full of inaccuracies, according to Clearspending, a new Sunlight site that tracks and illustrates just how broken the data is. We were deliberate in our approach when conducting this analysis, and we hope that by giving these problems the light of day, it might actually help get them fixed.

Think of Clearspending.com as a kind of scorecard that analyzes how well U.S. government agencies are reporting their spending data on USASpending.gov.

What Sunlight has found, and Clearspending shows in great detail, is that just under* $1.3 trillion in federal reporting data from 2009 is unreliable. The data inaccuracies we uncovered account for 70 percent of the total $1.9 trillion in government spending data reported in that year. Some of the numbers are too big, some are too small and some are missing completely, while other spending data entries don’t have the detail that’s required or were reported months later than the law demands.

When it comes to making data available, it has to be accurate. Federal agencies need to focus first on the quality of data they collect. If the data is unreliable, then the quality of websites they release — or the tools built upon it  — is irrelevant.

The government has known about the problems we’ve illustrated on Clearspending, and they say they’re working to fix it. But instead we have only gotten a series of redesigned websites, each one with data just as unreliable as the one before it.

There’s a tremendous amount of work left to do before Gov2.0 becomes a reality. These are not easy tasks, and certainly not glamorous ones. But these are the types of challenges that we must undertake if the promise of Gov2.0 is going to be realized.

*We updated this number - at launch it was just north of $1.3 trillion. You can see why here.

This Sunshine Week was a particularly successful (and busy!) time for Sunlight. We helped usher in new transparency legislation, launched a nation-wide grassroots campaign, launched a new design contest, joint led a crowdsourced search for government data, met with editorial boards and more. No wonder we’re all exhausted this Friday afternoon.

Here’s a more granular look at what we accomplished this week:

For Sunlight’s Communications team, Sunshine Week began early, with extensive outreach to editorial boards, reporters and columnists. This strategy paid off, as Sunday’s kick off of Sunshine Week by newspapers across the country included many mentions of Sunlight’s work. (You can read all of our mentions on our site -- and note we’re still updating the list.)

On Monday, John Wonderlich, Gabriela Schneider and I met with an editorial writer Washington Post to discuss initiatives we support that would further improve public access to government information including the Public Online Information Act.

That bill was introduced at a press conference on Tuesday (under a transparently clear sky) on Capitol Hill. I joined Rep. Steve Israel and the Personal Democracy Forum’s Andrew Rasiej (who is also an adviser to Sunlight) in announcing this landmark legislation. You can watch our press conference if you want to know more details about this groundbreaking legislation.

On Wednesday, Sunlight Labs launched our fourth official contest: Design for America. Part contest, part festival, the Design for America contest's intent is to inspire the design community to tell great stories about how our government works, what our government does, and what it could do. It's a contest as much about possibility as transparency, and with categories ranging from infographics to web design, there's plenty for all to compete.

Also on Wednesday, we joined the Center for Public Integrity in crowdsourcing our Data Mine project, a new online series identifying inaccessible or difficult to use information from the federal government. We need your tips on what federal government records, databases, and filings should be open to the public – join us to help so  you can continue to read about public government data that we still can’t access. Follow  Jim Morris’ posts on the Reporting Group site, like his most recent  report.

On Thursday, Sunlight’s Engagement team kicked off our new Public=Online campaign . Jake Brewer led a roundtable discussion at Google’s D.C. headquarters, with Jose Antonio Vargas from Huffington Post; Jim Harper from Cato Institute and WashingtonWatch.com; Ginny Hunt from Google’s Public Sector Lab and Ryan Hopkins of Public Square Project in Pittsburgh. Public=Online is a completely non-partisan campaign with the first goal of making government transparency a major issue in the upcoming mid-term elections. We hope you’ll sign the pledge and join us.

I traveled to Boston that day to meet with the Boston Globe editorial board, speak with fellows at the Berkman Center for Internet & Society at Harvard Law School, and at an MIT’s Center for Future Civic Media Forum.

Our Engagement team also organized a Sunshine Week Blogswarm, in which about 50 bloggers across the country pledged they would blog about the importance of transparency to them. Look for a "Local Sunlight" blog post early next week by Nisha Thompson with a roundup.

Today, John Wonderlich joined White House Special Counsel to the President for Ethics and Government Reform, Norm Eisen, and Jim Harper from Cato and WashingtonWatch.com (which I should note is a Sunlight grantee) at the OpenTheGovernment.org “Building Transparency” panel at the Center for American Progress.

Whew! What a week.

As a former Capitol Hill Communications Director, I can tell you that access to real-time information on what is happening on the Hill can make or break a successful advocacy campaign. Information is power, and the Sunlight Labs new Real Time Congress App for the iPhone gives users access to instantaneous in-the-know information in the palm of your hand. By pulling together RSS and XML feeds from the party policy committees, leadership offices, news outlets, bill texts and the alphabet soup of analysts (Think CBO, OMB, CRS et al.), the coders at the Labs have created a rich and valuable user experience for anyone who is interested in what is happening in Congress.

Forgive my enthusiasm for this new app, but it really is something special and elegant. Forget the fact that the platform will be expanded and new data sources will be added and the app will be expanded. I know that I am channeling my inner Don Draper here when I say that this new app brings me home again, and by home, I mean the Longworth House Office Building.

The Real Time Congress application for iPhone will keep journalists, Hill staffers, bloggers and interested citizens up to date on what is happening in Congress, in real-time. Its ease of use and sleek design promise that end users will continue to go back to the app for unfiltered information on Congress so they can make their own informed decisions on what is happening in the Capitol.

Our goal at the Sunlight Foundation is to change the way that citizens collect information about their government, and then help them to use that information to change the way they interact with their government. This new app shows how powerful new programs and smart phones can accomplish that goal. I’m just a little jealous of my former colleagues on the hill—I kind of wish I had this when I was working over there.

We're pretty proud of the app and it's free to you as the user. It is worth noting, however, that it wasn't free for us to create. It did take weeks of development, and so any contribution toward this application and all the others we hope to create in the future on your behalf is greatly appreciated.