Earlier this afternoon the Department of Justice and Attorney General Eric Holder celebrated Sunshine Week by highlighting the federal government's progress "in realizing the promise of the Freedom of Information Act." Holder and four additional speakers pointed out what they called positive steps taken in 2011 to reduce request backlogs, improve processes, and operate under a "presumption of openness." This positive news was tempered by today's Associated Press report that indicates that the federal government is still struggling with FOIA backlogs.

In touting the Department's accomplishments, Holder  looked toward the future and presented some improvements to FOIA currently being instituted by the DOJ. He announced  the DOJ will start posting monthly logs of FOIA requests made to senior leadership offices. The logs will "publicly identify the subject matter and disposition of each request" in an attempt to make it easier for people to locate information they are interested in. The department is also working on a new way for the public to submit and track FOIA requests to the DOJ's senior leadership online.

Additionally, the department is rolling out two new tools in an attempt to make FOIA.gov more responsive; a simplified government-wide search function and an integrated FOIA request process.

Four speakers from across the federal government joined Holder and touted the progress their offices made on FOIA issues

• Carolyn Colvin, Deputy Commissioner at the Social Security Administration, spoke to the SSA's successful implementation of a FOIA Process Evaluation Working Group, which helped improve efficiency.
• Austin Schlick, General Counsel and Chief FOIA Officer at the FCC, highlighted the overhauled FCC website and greater online access to Commission information.
• Darren Ash, CIO and Chief FOIA Officer at the Nuclear Regulatory Commission, described the NRC's efforts to deal with a surge of FOIA requests following the tsunami and nuclear crisis in Japan last year.
• Robert Howarth, Deputy Director of Correspondence, Document Production and FOIA Management at the Department of Interior, detailed a reorganization of FOIA leadership at the DOI.

Attorney General Holder's full remarks can be read here.

Update: The National Security Archive has responded to Holder's speech. They strongly criticize the Attorney General for citing discredited statistics in his remarks. They also note that the DOJ has attempted to issue reductive regulations, waged a "war on leakers", and increasingly relied on several exemptions throughout Holder's tenure. The National Security Archive recently awarded the Department of Justice their Rosemary Award for worst open government performance by a federal agency in 2011.

Policy Fellow Matt Rumsey wrote this post. 

The News Without Transparency: Records not so Open with Obama

The Department of Justice’s recently proposed changes for the way it executes Freedom of Information Act requests have inspired a flurry of media attention from various news outlets and criticism from government watchdogs.

The groups criticizing the DOJ, which includes the Sunlight Foundation, are concerned that the new rules are too restrictive and threaten the federal law’s usefulness and fairness to the regular citizens and journalists it is meant to serve.

Here at Sunlight, we compared the new and old regulations, and discussed how the changes would be a huge step back for transparency. But we’re not the only organization writing about FOIA problems and setbacks. The backlog of FOIA requests and agencies taking much longer than allowed by the statute to fill requests are examples of the frequently discussed issues over the past few years.

Based on data gathered by government entities responsible for collection of FOIA statistics, and despite the presidential promise to do better, it appears things have only gotten worse. In March 2010 the Associated Press assessed how well President Obama's promises to comply more fully with FOIA had been carried out after a year.

The article found that agencies were more frequently citing exemptions to avoid complying with requests. Similarly, in July 2011 iWatch News vented frustration with the State Department for neglecting to respond to FOIA requests for four years.

These articles, as well as others that look at the administration's compliance with FOIA, frequently begin with holding agency practices to the standard set by President Obama's memo to the heads of executive departments and agencies. The memo states, "The Freedom of Information Act should be administered with a clear presumption: In the face of doubt, openness prevails." The text of this memo is available from the Briefing Room on WhiteHouse.gov. It is located under "Presidential Memoranda" under January 2009.

In general, agencies are required to submit annual reports detailing their interaction with FOIA requests. FOIA.gov links to At-a-Glance reports that highlight the major findings from each agency. FOIA.gov also allows you to search the data submitted as part of agency reports and create acustom-made report. The Office of Information Policy also generates specific reports to emphasize interesting FOIA statistics.

The AP article states that agencies cited the "deliberative process" exemption - one that the president's memo specifically directed agencies to avoid using so frequently - 70,779 times in 2009. This is up from the 47,395 times it was cited under George W. Bush in 2008.

The article also states that agencies overall cited exemptions at least 466,872 times in 2009, again up from 312,683 times in 2008. Meanwhile requests are declining, down by 11 percent - 493,610 in 2008 to 444,924 in 2009.

The AP also claims that the "majority of agencies - 12 out of the 17, or 70 percent of those surveyed - increased FOIA requests granted in full, in part or both."

On what could be a positive note for the administration,  the AP notes that the number of backlogged requests had dropped from 124,019 in 2008 to 67,764 in 2009.

These facts and statistics about the number of FOIA requests filed and completed can be found using the "Advanced Report" function on the FOIA website. This function allows you to select the specific data you would like included in your report as well as which agencies you would like included in a comparison. Unlike the basic report function, "all agencies" is an option using this tool. Specific data choices include exemptions, requests, appeals, processing time, requests granted, and backlog.

Much of this information could also be obtained by looking individually at each agency's annual report. The AP article only looked at 17 agencies, but the FOIA data tool allows you to look at many more if you so choose.

The iWatch News story states that for the State Department, "the median response for complex FOIA requests is 228 days." The State Department makes available its annual FOIA reports from 1998 through the present. The 2010 report provides the statistic used in the article.

The story also states that for fiscal year 2012 the State Department has requested "$166,000 in new money to depend the department in FOIA-related lawsuits." The department's budget requests are available from 2002 through the present. The 2012 "Department of State Operations Congressional Budget Justification" includes the numbers cited in the article.

---

"The News Without Transparency" shows you what the news would look like without public access to information. Laws and regulations that force the government to make the data it has publicly available are absolutely vital, along with services that take that raw data and make it easy for reporters to write sentences like the ones we've redacted in the piece above. If you have an article you'd like us to put through the redaction machine, please send us an email at mbuck@sunlightfoundation.com.

This fourth of July marks the 45th anniversary America's freedom of information law. FOIA transformed our world by giving teeth to the public's right to know. It made government prove why public information shouldn't be disclosed, instead of forcing people to justify why it should be. The internet age has brought another revolution, transforming how we expect information will be available to us. But our freedom of information laws have slowly calcified through a combination of inertia, bureaucratic reluctance, and lack of funding.

Just as FOIA of the 1960s embraced disclosure upon demand, it's time to reinvent our freedom of information laws for the internet age and embrace affirmative disclosure -- where government information is routinely published online, in real time, and in machine readable formats.

We have already seen some baby-steps in this direction. For example:

• The 1996 Electronic Freedom of Information Act Amendments (or E-FOIA) required agencies to publish records that they expected would be the subject of multiple requests in an "electronic reading room," and instructed agencies to make records available in electronic format when requested.
• The Obama Administration created Data.gov, an online repository of "high value, machine readable" datasets generated by the Executive branch, and launched the Open Government Directive, which (among other things) encouraged agencies to release more information to the public. (These initiatives have been subject to significant funding cuts.)
• The Office of Government Information Services Office of Information Policy at the Department of Justice* recently launched FOIA.gov, a one-stop shop to see how well agencies are staying on top of their FOIA requests.
• Nearly all agencies have websites where limited information is disclosed about their operations and activities.

The scope of the public access is much larger than these initiatives can address. The public's information needs spans the scope of government activity, from census data to spending data, from policy and position papers to economist forecasts, from maps to legal information, and much more. It is worth highlighting four particularly clever ideas for next steps as examples of where things could go.

The Public Online Information Act (or POIA) is pending legislation, originally introduced by Rep. Israel and Sen. Tester, that would require executive branches agencies to publish all publicly available information in the Internet in a timely fashion and in user-friendly formats, and create an advisory committee to help develop government-wide Internet publication standards. Like FOIA, POIA allows the public to go to court if agencies fail to comply.

In the UK, the website WhatDoTheyKnow.com makes many government information requests publicly available. You submit your FOIA-like request through the website, they submit the request to the appropriate agency and publish the answer online. This reduces the number of duplicate requests while making the sum of information released available to everyone. (There's a similar effort to crowd-source state-level freedom of information requests in the US called MuckRock. Disclosure: Sunlight gave funding to MuckRock.)

Data quality is a big deal. The ten open data principles lay out a means to evaluate the extent to which data is open and accessible to the public. Similarly, efforts like ClearSpending mash up government data to evaluate the accuracy of information reported by the government in the first place. It's difficult for the public to make use of information that is badly formatted or inaccurately reported.

It's not enough to make sure that the data reported by the government is accurate. We must also ensure that government identifies and reports on all the data that it has. The government must audit its holdings and build an index of what it has and who is responsible for maintaining the information. The President's recent Memo on Regulatory Compliance, for example, addresses how private entities disclose information to the agencies that regulate them and how that data is reviewed, shared inside government, and (whenever appropriate) made available to the public.

The public's need to access government-held information is as old as our libraries, post offices, and government publications. It's time to make our public information available online.

• Corrected to reflect the right agency. OGIS mediates FOIA disputes and reviews agency compliance with FOI; OIP launched FOIA.gov.

Today the Sunlight Foundation is proud to unveil Sarah's Inbox, our attempt to make Sarah Palin's recently released email records easier to use with a searchable function and an interface similar to Gmail. It builds on Elena's Inbox, our wildly popular project launched almost exactly one year ago that took the email data of Supreme Court justice Elena Kagan released by the Clinton Library and made it more accessible online.

Sarah's Inbox allows users to view the more than 14,000 emails from Sarah Palin's tenure as Governor of Alaska with familiar sorting functions. You can go page by page starting from the most recent emails or, most importantly, search. To help direct folks to interesting items, try some of our sample searches, star emails for later viewing or view the most starred emails by all users.

The project started after we were again approached by folks on Twitter and the Sunlight Labs list (join!) to take this ugly data and add the Sunlight secret sauce to make it user friendly. Initially we were cautious because the cast of characters who directly obtained the data included the likes of the New York Times, ProPublica, Mother Jones and MSNBC.com. We spoke with ProPublica and they encouraged us to take a stab at fashioning our own tool, so we borrowed their data and went to work. Sarah's Inbox would not be possible if not for the great people at Crivella West to gather, lift, scan and pay for all this data.

Like Elena's Inbox, Sarah's Inbox faced staggering issues of data quality because government officials continue to release digital files as hideous printouts requiring a laborious and error-ridden optical character recognition (OCR) pass over. You will notice that many of the emails are garbled, incomplete or contain odd characters - please keep in mind that we did the best with what we had and are not responsible for the content. Due to the programmatic nature of the tools used to build this site, we recommend checking any research effort against the source files.

Disclaimers aside, please enjoy Sarah's Inbox and tweet interesting items you find with #sarahsinbox.

Sunshine Week starts with a new report from the National Security Archive and the Knight Foundation that finds only 49 of 90 agencies have adopted 'concrete steps' to improve their responsiveness to Freedom of Information Act requests. This is incredibly disheartening, though an improvement on the numbers from last year's study that found 13 of 90 agencies following up. The agencies' failure to meet even the administration's low bar is unacceptable. The two 'concerte steps' are simply updating the language in FOIA training documents to presume openness and to assess whether resources for compliance are adequate.

The report stems from Obama's Executive Order that called for greater FOIA openness [link], one of the first official acts he made as President, a follow-up memo from Attorney General Eric Holder detailing the new principles on FOIA [pdf link] and another memo a year later from former Chief of Staff Rahm Emanuel and former Counsel to the President Bob Bauer asking agencies to please take the baby steps previously promised [pdf link].

“At this rate, the president’s first term in office will be over by the time federal agencies do what he asked them to do on his first day in office,” commented Eric Newton, senior adviser to the president at the John S. and James L. Knight Foundation, which funded the study. “Freedom of information laws exist to help all of us get the information we need for this open society to function. Yet government at all levels seems to have a great deal of trouble obeying its own transparency laws.”

Modeled after the California Sunshine Survey and subsequent state “FOI Audits,” the Archive’s series of Knight Open Government Surveys started in 2002 and use open government laws to test whether or not agencies are obeying those same laws. Recommendations from previous Knight Open Government Surveys led directly to laws and executive orders which have: set explicit customer service guidelines, mandated FOIA backlog reduction, assigned individualized FOIA tracking numbers, forced agencies to report the average number of days needed to process requests, and revealed the (often embarrassing) ages of the oldest pending FOIA requests.

Among the varying responses from agencies, the most stunning result was the U.S. Postal Service saying it had "no responsive records" and never even received the Emanuel-Bauer memo! Below is a chart of the ratings of each agency:

American University's Investigative Reporting Workshop began a new blog yesterday entitled 'Exemption 10' in reference to the unwritten tenth exemption to rejecting a Freedom of Information Act request. Sunlight's Reporting Group team often runs into frustrating FOIA exemptions in their investigations and even compliance can be galling, such as the 3.5" floppy disk Sunlight once received. (We are not making this up.)

This new blog will cover open government issues and we're excited to see a new outlet dedicated to this work. Here's an excerpt from the inaugural blog post:

We expect to write about case studies (we’d love to hear from you), to discuss court cases dealing with FOIA, to highlight and aggregate coverage from other sources, and to be involved with the FOIA community in Washington and beyond.

And just so there is no confusion, Exemption 10 and the Investigative Reporting Workshop will be strong advocates for government openness. It is the one subject where we believe it is not only appropriate, but also necessary, for news organizations to be deeply involved in the shaping of public policy.

We’d like to help create an even stronger community around this vital issue, which means we’d like your comments, suggestions, case studies and ideas. You can follow us on Twitter @irworkshop or send us email.

A good place to start would be highlighting the Senate's e-filing bill and possibilities for lobbying reform.

This is pretty disheartening. The Obama administration is continuing to use the same arguments the Bush administration used to block disclosure of visitor logs, even in limited cases. Both msnbc.com and, in a more limited request, Citizens for Responsibility and Ethics in Washington (CREW) were denied access to the logs by the Secret Service. The Service claimed that the logs are presidential and not agency records, thus not subject to the Freedom of Information Act (FOIA). This continues the policy implemented by the Bush White House as they tried to block disclosure of visits to the White House by leaders of the Religious Right and, later, corrupt lobbyist Jack Abramoff.

Despite court rulings requiring the disclosure of the logs, the Bush administration continued to stonewall and appeal the decisions. The Obama adminsitration has continued those appeals, while also saying the policy is "under review." This looks like another example of the Executive Branch refusing to reliquinsh unnecessary secrecy that it seized in the face of a court order. This is also far more harmful to Executive Branch transparency than the failure to post bills on WhiteHouse.gov for five days. Hopefully, when the administration says that the policy is "under review" they actually mean it and this isn't like the Supreme Leader of Iran saying that he will investigate election fraud.

Nick Troiano at SocialGovernment.com has an interesting and important post about government transparency, the 2.0 version. Nick was reflecting from a discussion featured by the National Conference on Citizenship titled “In Transparency, We Trust?

He says “transparency 1.0” was government opening up its data for citizens to see. That age dawned in 1966 when Congress passed and President Lyndon Johnson signed into law the Freedom of Information Act.  Another transparency 1.0 manifestation was when government entities started setting up Web sites. But the communication was all one way, flowing from government to the citizens. But as Nick points out, our expectations of transparency in government have changed and now include participation.” This is the essence of “transparency 2.0,” where “a window between the people and their government” no longer will suffice. We need to have the ability to reach through the window and “fiddle around.” Observation is fine, but participation is now key. Nick notes that progress has been made toward the participatory aspects of transparency 2.0. He lists congressional lawmakers communicating with their constituents via Twitter and YouTube, President Obama’s online town hall meeting and Sunlight’s PublicMarkup.org, were citizens collaborated together write and comment on sample legislation. Lisa Rosenberg, Sunlight’s government affairs consultant, participated in the NCoC discussion, and said that the goal of transparency should be to open up discussions, improve the deliberative process, and help our democracy live up to its potential.  If government conducted its business online and in real time a more thoughtful, deliberative conversation between elected officials and the public could be created. This, in turn, would result in better public policy, more careful monitoring of the public purse, and more trust in government. Nick predicts transparency 2.0 will be the backbone of a more social, and thus, responsive government. Empowered citizens working with public officials to make informed decisions are benefits that transparency 2.0 promises to deliver.

Since the Memorandum on FOIA and Government Transparency are not yet posted at WhiteHouse.gov, I thought I would post them both on the Sunlight's blog. As you can imagine, we love the use of the Justice Brandeis' quote in the FOIA Memoradum.

MEMORANDUM FOR THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES

SUBJECT: Freedom of Information Act

A democracy requires accountability, and accountability requires transparency. As Justice Louis Brandeis wrote, "sunlight is said to be the best of disinfectants." In our democracy, the Freedom of Information Act (FOIA), which encourages accountability through transparency, is the most prominent expression of a profound national commitment to ensuring an open Government. At the heart of that commitment is the idea that accountability is in the interest of the Government and the citizenry alike.

The Freedom of Information Act should be administered with a clear presumption: In the face of doubt, openness prevails. The Government should not keep information confidential merely because public officials might be embarrassed by disclosure, because errors and failures might be revealed, or because of speculative or abstract fears. Nondisclosure should never be based on an effort to protect the personal interests of Government officials at the expense of those they are supposed to serve. In responding to requests under the FOIA, executive branch agencies (agencies) should act promptly and in a spirit of cooperation, recognizing that such agencies are servants of the public.

All agencies should adopt a presumption in favor of disclosure, in order to renew their commitment to the principles embodied in FOIA, and to usher in a new era of open Government. The presumption of disclosure should be applied to all decisions involving FOIA.

The presumption of disclosure also means that agencies should take affirmative steps to make information public. They should not wait for specific requests from the public. All agencies should use modern technology to inform citizens about what is known and done by their Government. Disclosure should be timely.

I direct the Attorney General to issue new guidelines governing the FOIA to the heads of executive departments and agencies, reaffirming the commitment to accountability and transparency, and to publish such guidelines in the Federal Register. In doing so, the Attorney General should review FOIA reports produced by the agencies under Executive Order 13392 of December 14, 2005. I also direct the Director of the Office of Management and Budget to update guidance to the agencies to increase and improve information dissemination to the public, including through the use of new technologies, and to publish such guidance in the Federal Register.

This memorandum does not create any right or benefit, substantive or procedural, enforceable at law or in equity by any party against the United States, its departments, agencies, or entities, its officers, employees, or agents, or any other person.

The Director of the Office of Management and Budget is hereby authorized and directed to publish this memorandum in the Federal Register.

BARACK OBAMA

Lots of folks are starting to think about the transition to a new Administration.  We know of at least 2 dozen such efforts thus far.  And so we'll blog about them as they start to get released. 

POGO, the Project on Government Oversight, just released a list of recommendations of major reforms they believe the new presidential administration should adopt to make the federal government more effective, accountable, open, and honest. And we sure like their list! POGO sent their list to both the McCain and Obama transition teams.

As Mandy Smithberger writes at the POGO blog, they are advising the transition teams to making agency missions more modern and relevant, keeping government's role federalized, protecting whistleblowers, stopping the revolving door between government and private industry, strengthen the Freedom of Information Act and other transparency reforms, and other reforms to provide better government oversight.

Our friends at POGO say their recommendations, if implemented, will make the federal government work more efficiently, as well as shrink the cost of government. As the report states, "the initial costs of these reforms will be more than offset by the longterm savings for the taxpayer."