The Cyber Intelligence Sharing and Protection Act (CISPA) passed the House by a comfortable margin last week despite loud opposition from privacy groups, a veto threat from the White House, and uncertain prospects in the Senate. Lawmakers made several changes to the bill aimed at easing privacy concerns. Unfortunately, a provision that should give transparency advocates pause not only survived, but is spreading to other cybersecurity legislation.

When CISPA was originally introduced in the 112th Congress it contained language that would effectively exempt all information about "cyber threats" shared via the bill from the Freedom of Information Act. That provision survived in the version of CISPA that passed the House last week, and similar language has worked its way into another piece of  legislation, the SECURE IT Act, introduced earlier this month.

Wholesale exemptions for "cyber threat information"  will prevent public oversight and deny citizens and watchdogs the ability to understand how the government and businesses communicate about and respond to cyber threats. The most sensitive information that would be shared through these bills is already protected from disclosure through existing FOIA exemptions. It is hard to see a compelling reason to subvert the FOIA altogether when it comes to cybersecurity.

Privacy advocates are concerned that personal information will be subject to over-sharing and misuse. Without access to rights provided by the FOIA there will be no way to hold those in power accountable if they are collecting too much information or misusing the data they obtain.

The Freedom of Information Act is a cornerstone for public oversight of government activity. Any change to the law deserves a vigorous and open debate.

CISPA and the SECURE IT Act give government officials broad new powers and the current FOIA provisions provide them with blanket protection from public scrutiny. These new, overly broad exemptions  are unnecessary and should not be passed into law.

Last year we called the Cyber Intelligence Sharing and Protection Act (CISPA), H.R. 3523 in the 112th Congress, "terrible on transparency," because of a provision that would exempt a broad and poorly defined swath of information from the Freedom of Information Act.

CISPA is back in the 113th Congress and not much has changed. According to OpenTheGovernment.org, the bill still contains language that would exempt all "cybersecurity threat information" shared under the act from the FOIA. This is a dangerous and unnecessary precedent to set. As we noted last year, implementing a broad exception for poorly defined information without a public hearing "is irresponsible and should be opposed."

The Congressional debate over CISPA is currently taking place behind the closed doors of the House Select Committee on Intelligence. Chairman Mike Rogers has announced his committee's intent to mark up and vote on CISPA this month and a committee spokesperson has suggested that it will meet behind closed doors next week to consider the bill.  The committee does not appear to be planning any public hearing or vote.

Along with 40 other groups, Sunlight has signed a letter urging the House Permanent Select Committee on Intelligence to hold any markup of CISPA in the open.

The committee should consider CISPA in the light of day. CISPA has the potential to affect the lives of every American and has drawn criticism from numerous privacy groups as well as the White House. Legislation this critical, and with this much potential to hinder the public's ability to hold their government accountable, deserves to be debated in full public view, not hidden behind closed doors.

You can follow CISPA using Scout, Sunlight's legislative tracking tool.

The Freedom of Information Act (FOIA) is a foundational law that guarantees US citizens the right to request and receive information from federal government agencies, with some relatively narrow exceptions. In a move reminiscent of the the Department of Justice's attempt to defang FOIA, the Federal Mediation and Conciliation Service—which mediates labor disputes around the United States—just finalized some changes to their FOIA rules that appear to raise the price and difficulty for citizens requesting information.

For example, current law offers a straightforward public interest fee waiver or discount:

Documents are to be furnished without charge or at reduced levels if disclosure of the information is in the public interest; that is, because it is likely to contribute significantly to public understanding of the operations or activities of the Government and is not primarily in the commercial interest of the requester.

The new rule changes this to explicitly say that this determination is up to FMCS, and to leave room for the FMCS to deny a waiver even if the request is judged to be in the public interest (emphasis mine):

Documents may be furnished without charge or at reduced levels if FMCS determines that disclosure of the information is in the public interest; that is, because it is likely to contribute significantly to public understanding of the operations or activities of the Government and is not primarily in the commercial interest of the requester.

There is a raft of other changes that raise fees, increase standard processing times, allow the agency to label a request as "complex" and put it into a longer queue, and allow indefinite delays to processing.

Interestingly, two policies are being removed entirely - one that says dispute notices are disclosable, and one that mandates that the agency report to Congress what requests were denied and why. The regulation states these are removed because "they are neither required by law nor necessary to interpret the law." It's not clear whether the agency now believes that the original legislative statute is sufficient to guarantee these policies, or whether the agency no longer believes they are required to implement them.

These regulations were first proposed in 1999, then re-proposed in 2007, and only made final on Tuesday, November 6th. The second and final versions each say that no public comments were received on the version before it.

If you'd like to analyze what this rule changes, the easiest way I've found is comparing the rule's web version (or public inspection PDF) on FederalRegister.gov to Cornell's online Code of Federal Regulations. To follow ongoing changes to FOIA around the government, I recommend using Scout, a tool we built to do just that, and how this particular rule came to our attention.

One of the most precise ways to follow an issue you care about is to identify where it sits in the law, and watch how people are attempting to change it for good or ill.

For example, as a transparency organization, we follow the Freedom of Information Act quite closely, which is codified primarily as 5 USC § 552. Up to now, we've been following it in Scout by searching for specific text strings, such as "5 U.S.C. 552" and "section 552 of title 5".

But setting up multiple searches is inconvenient, and neither of these catch results that cite subsections - for example, some bills affecting exemptions to FOIA will cite "section 552(b)(3) of title 5". Setting up searches for each subsection, in each common citation format, isn't practical.

So we've made Scout's citation searching much smarter: now, if you search for "5 usc 552", or "section 601 of title 5", you'll see that it returns results matching a variety of formats and subsections in bills and regulations, all at once.

This makes keeping up with proposed changes to laws much more reliable and complete, and we hope you find it useful.

Under the hood

Technically, this is accomplished by pre-scanning all bills and regulations for any kind of citation to the US Code. We do this by running the text through Citation, a JavaScript library for citation extraction. It finds citations, pulls out an excerpt, breaks the citation down into its component parts, and assigns it a unique ID. Right now, it's using just two regular expressions to detect citations, but it seems to catch a great many of them.

On Scout's end, it looks to see if the search query looks like a US Code citation, converts it to the same kind of unique ID that Citation.js uses, and asks for any bills or regulations which have that particular US Code section ID associated with it. So, though the search results in Scout look similar to a full text search, it's actually just filtering on an ID, and displaying the excerpts we already extracted from each bill and regulation.

Future

There's lots left to do here - there are some complicated US Code citations that aren't yet caught, and we'd like to expand to other citation types (such as the CFR). Scout also only does this kind of special search for "simple phrase" searches, so it's currently not possible to filter on US Code citations and another search term - so we'll add that before long.

I expect that we'll be publishing our extracted citation data in bulk, eventually. However, right now it still misses enough complex citations that it may not be useful yet as a canonical set of legal links, even though it's clearly useful in an integrated search context.

But the code to extract them, Citation, is open source, and even slightly documented. Contributions are helpful and welcome - we hope that it can grow into something that everyone can use, so that this work doesn't have to be done again.

Earlier this afternoon the Department of Justice and Attorney General Eric Holder celebrated Sunshine Week by highlighting the federal government's progress "in realizing the promise of the Freedom of Information Act." Holder and four additional speakers pointed out what they called positive steps taken in 2011 to reduce request backlogs, improve processes, and operate under a "presumption of openness." This positive news was tempered by today's Associated Press report that indicates that the federal government is still struggling with FOIA backlogs.

In touting the Department's accomplishments, Holder  looked toward the future and presented some improvements to FOIA currently being instituted by the DOJ. He announced  the DOJ will start posting monthly logs of FOIA requests made to senior leadership offices. The logs will "publicly identify the subject matter and disposition of each request" in an attempt to make it easier for people to locate information they are interested in. The department is also working on a new way for the public to submit and track FOIA requests to the DOJ's senior leadership online.

Additionally, the department is rolling out two new tools in an attempt to make FOIA.gov more responsive; a simplified government-wide search function and an integrated FOIA request process.

Four speakers from across the federal government joined Holder and touted the progress their offices made on FOIA issues

• Carolyn Colvin, Deputy Commissioner at the Social Security Administration, spoke to the SSA's successful implementation of a FOIA Process Evaluation Working Group, which helped improve efficiency.
• Austin Schlick, General Counsel and Chief FOIA Officer at the FCC, highlighted the overhauled FCC website and greater online access to Commission information.
• Darren Ash, CIO and Chief FOIA Officer at the Nuclear Regulatory Commission, described the NRC's efforts to deal with a surge of FOIA requests following the tsunami and nuclear crisis in Japan last year.
• Robert Howarth, Deputy Director of Correspondence, Document Production and FOIA Management at the Department of Interior, detailed a reorganization of FOIA leadership at the DOI.

Attorney General Holder's full remarks can be read here.

Update: The National Security Archive has responded to Holder's speech. They strongly criticize the Attorney General for citing discredited statistics in his remarks. They also note that the DOJ has attempted to issue reductive regulations, waged a "war on leakers", and increasingly relied on several exemptions throughout Holder's tenure. The National Security Archive recently awarded the Department of Justice their Rosemary Award for worst open government performance by a federal agency in 2011.

Policy Fellow Matt Rumsey wrote this post. 

"The News Without Transparency" shows you what the news would look like without public access to information. Laws and regulations that force the government to make the data it has publicly available are absolutely vital, along with services that take that raw data and make it easy for reporters to write sentences like the ones we've redacted in the piece above. If you have an article you'd like us to put through the redaction machine, please send us an email at mbuck@sunlightfoundation.com.

This fourth of July marks the 45th anniversary America's freedom of information law. FOIA transformed our world by giving teeth to the public's right to know. It made government prove why public information shouldn't be disclosed, instead of forcing people to justify why it should be. The internet age has brought another revolution, transforming how we expect information will be available to us. But our freedom of information laws have slowly calcified through a combination of inertia, bureaucratic reluctance, and lack of funding.

Just as FOIA of the 1960s embraced disclosure upon demand, it's time to reinvent our freedom of information laws for the internet age and embrace affirmative disclosure -- where government information is routinely published online, in real time, and in machine readable formats.

We have already seen some baby-steps in this direction. For example:

• The 1996 Electronic Freedom of Information Act Amendments (or E-FOIA) required agencies to publish records that they expected would be the subject of multiple requests in an "electronic reading room," and instructed agencies to make records available in electronic format when requested.
• The Obama Administration created Data.gov, an online repository of "high value, machine readable" datasets generated by the Executive branch, and launched the Open Government Directive, which (among other things) encouraged agencies to release more information to the public. (These initiatives have been subject to significant funding cuts.)
• The Office of Government Information Services Office of Information Policy at the Department of Justice* recently launched FOIA.gov, a one-stop shop to see how well agencies are staying on top of their FOIA requests.
• Nearly all agencies have websites where limited information is disclosed about their operations and activities.

The scope of the public access is much larger than these initiatives can address. The public's information needs spans the scope of government activity, from census data to spending data, from policy and position papers to economist forecasts, from maps to legal information, and much more. It is worth highlighting four particularly clever ideas for next steps as examples of where things could go.

The Public Online Information Act (or POIA) is pending legislation, originally introduced by Rep. Israel and Sen. Tester, that would require executive branches agencies to publish all publicly available information in the Internet in a timely fashion and in user-friendly formats, and create an advisory committee to help develop government-wide Internet publication standards. Like FOIA, POIA allows the public to go to court if agencies fail to comply.

In the UK, the website WhatDoTheyKnow.com makes many government information requests publicly available. You submit your FOIA-like request through the website, they submit the request to the appropriate agency and publish the answer online. This reduces the number of duplicate requests while making the sum of information released available to everyone. (There's a similar effort to crowd-source state-level freedom of information requests in the US called MuckRock. Disclosure: Sunlight gave funding to MuckRock.)

Data quality is a big deal. The ten open data principles lay out a means to evaluate the extent to which data is open and accessible to the public. Similarly, efforts like ClearSpending mash up government data to evaluate the accuracy of information reported by the government in the first place. It's difficult for the public to make use of information that is badly formatted or inaccurately reported.

It's not enough to make sure that the data reported by the government is accurate. We must also ensure that government identifies and reports on all the data that it has. The government must audit its holdings and build an index of what it has and who is responsible for maintaining the information. The President's recent Memo on Regulatory Compliance, for example, addresses how private entities disclose information to the agencies that regulate them and how that data is reviewed, shared inside government, and (whenever appropriate) made available to the public.

The public's need to access government-held information is as old as our libraries, post offices, and government publications. It's time to make our public information available online.

• Corrected to reflect the right agency. OGIS mediates FOIA disputes and reviews agency compliance with FOI; OIP launched FOIA.gov.

Today the Sunlight Foundation is proud to unveil Sarah's Inbox, our attempt to make Sarah Palin's recently released email records easier to use with a searchable function and an interface similar to Gmail. It builds on Elena's Inbox, our wildly popular project launched almost exactly one year ago that took the email data of Supreme Court justice Elena Kagan released by the Clinton Library and made it more accessible online.

Sarah's Inbox allows users to view the more than 14,000 emails from Sarah Palin's tenure as Governor of Alaska with familiar sorting functions. You can go page by page starting from the most recent emails or, most importantly, search. To help direct folks to interesting items, try some of our sample searches, star emails for later viewing or view the most starred emails by all users.

The project started after we were again approached by folks on Twitter and the Sunlight Labs list (join!) to take this ugly data and add the Sunlight secret sauce to make it user friendly. Initially we were cautious because the cast of characters who directly obtained the data included the likes of the New York Times, ProPublica, Mother Jones and MSNBC.com. We spoke with ProPublica and they encouraged us to take a stab at fashioning our own tool, so we borrowed their data and went to work. Sarah's Inbox would not be possible if not for the great people at Crivella West to gather, lift, scan and pay for all this data.

Like Elena's Inbox, Sarah's Inbox faced staggering issues of data quality because government officials continue to release digital files as hideous printouts requiring a laborious and error-ridden optical character recognition (OCR) pass over. You will notice that many of the emails are garbled, incomplete or contain odd characters - please keep in mind that we did the best with what we had and are not responsible for the content. Due to the programmatic nature of the tools used to build this site, we recommend checking any research effort against the source files.

Disclaimers aside, please enjoy Sarah's Inbox and tweet interesting items you find with #sarahsinbox.

Sunshine Week starts with a new report from the National Security Archive and the Knight Foundation that finds only 49 of 90 agencies have adopted 'concrete steps' to improve their responsiveness to Freedom of Information Act requests. This is incredibly disheartening, though an improvement on the numbers from last year's study that found 13 of 90 agencies following up. The agencies' failure to meet even the administration's low bar is unacceptable. The two 'concerte steps' are simply updating the language in FOIA training documents to presume openness and to assess whether resources for compliance are adequate.

The report stems from Obama's Executive Order that called for greater FOIA openness [link], one of the first official acts he made as President, a follow-up memo from Attorney General Eric Holder detailing the new principles on FOIA [pdf link] and another memo a year later from former Chief of Staff Rahm Emanuel and former Counsel to the President Bob Bauer asking agencies to please take the baby steps previously promised [pdf link].

“At this rate, the president’s first term in office will be over by the time federal agencies do what he asked them to do on his first day in office,” commented Eric Newton, senior adviser to the president at the John S. and James L. Knight Foundation, which funded the study. “Freedom of information laws exist to help all of us get the information we need for this open society to function. Yet government at all levels seems to have a great deal of trouble obeying its own transparency laws.” Modeled after the California Sunshine Survey and subsequent state “FOI Audits,” the Archive’s series of Knight Open Government Surveys started in 2002 and use open government laws to test whether or not agencies are obeying those same laws. Recommendations from previous Knight Open Government Surveys led directly to laws and executive orders which have: set explicit customer service guidelines, mandated FOIA backlog reduction, assigned individualized FOIA tracking numbers, forced agencies to report the average number of days needed to process requests, and revealed the (often embarrassing) ages of the oldest pending FOIA requests.

Among the varying responses from agencies, the most stunning result was the U.S. Postal Service saying it had "no responsive records" and never even received the Emanuel-Bauer memo! Below is a chart of the ratings of each agency: