Earlier this week, appropriators held a hearing on funding for the legislative agencies that make government information available to the public.  Three organizations, the Sunlight Foundation, the Participatory Politics Foundation, and PopVox, filed comments on the importance of making legislative information directly available to the public as a downloadable database, instead of item-by-item, which is the current practice.

The Sunlight Foundation testified on this topic (a.k.a. "bulk access")  last year, and has sketched out some interesting new tools that it could empower. But of course, one major use would be to strengthen the already fantastic services available at OpenCongress and GovTrack, while supporting additional innovations.

Progress on bulk access has been slow. Several years ago, the Congress required the Library of Congress and others to examine the issue, but these agencies have dragged their heels and -- as far as we know -- have failed to finish that analysis. Sunlight's comments are available below.

Sunlight Foundation Bulk Access to THOMAS Testimony Leg Approps 2012-02-06

 

This Tuesday, there will be hearing on budgets for the Library of Congress, the Government Printing Office, the Government Accountability Office, and the Congressional Budget Office. It's too bad that the public won't have a real opportunity to learn about these important agencies, as the meeting is not expected to be webcast by the committee, and (if I remember correctly) the hearing room is so tiny that few if any members of the public will be able to attend.

That's too bad, especially because this is the first opportunity to hear firsthand how last year's budget cuts have affected agencies' abilities to do their jobs, and learn about agency and congressional priorities for the upcoming year. It's also the first time we'll hear from the new acting Public Printer  (the head of GPO); and perhaps the newly appointed head of the Congressional Research Service will be presented and introduced by the Librarian of Congress.

Only the House and Senate Legislative Appropriations Committees regularly hold annual public hearings on the workings of these agencies; the oversight committees (Committee on House Administration and Senate Rules) generally do not, and the Joint Committee on the Library and Joint Committee on Printing no longer holds substantive meetings in public.

The new House rules require that all committees provide "audio and video coverage of each hearing or meeting" that "allows the public to easily listen ... and view the proceedings" "to the maximum extent practicable." All of the House committees have at least one hearing room that is equipped with a camera, and the House Recording Studio will provide a camera upon a committee's request. Unfortunately, this hearing is being held in a room without a camera, and I've been informed that the Committee has not requested one. The Appropriations Committee has not scheduled any other hearings for Tuesday, so the room with the pre-positioned camera should be available.

We ran into this problem last year, when the Committee's justification for holding the meeting in the same  tiny, camera-less room (HT-2) was that it was more convenient to hold the hearing in the Capitol than in one of the legislative buildings. Even if convenience were more important than  the public access rule, the House Recording Studio could still provide a camera, and there are rooms in the newly constructed $600+ million Capitol Visitor Center (i.e. in the Capitol) that already have cameras installed. We would send a video crew ourselves, but only organizations accredited by the House Radio-Television Correspondents' Gallery can ask permission from the Committee to record the event, and the Sunlight Foundation doesn't qualify for membership.

Another change from last year is that members of the public are not invited to speak at the hearing, although they may submit written comments. Along with several others, I took the opportunity to speak last year, where I called for bulk access to THOMAS data and public access to CRS reports. I will submit comments for the record, but written comments are much less effective than speaking directly to the Members of Congress. It's too bad, especially because one of the major lessons of last Thursday's House Legislative Data and Transparency Conference  is that the Library of Congress and GPO have apparently been ignoring their legal obligation to make progress on public access to bulk data. Ironically, it was this very Committee that imposed the obligation upon them in the first place, 3 years ago.

As with everything in Congress, things could still change for Tuesday's hearing -- its time, date, location, and whether it will webcast or covered by the media. I plan on attending, and if I can make it into the room, I'll post an update.

The following are my notes for remarks I delivered at the House Legislative Data and Transparency Conference on February 2, 2012. They've been updated to include hyperlinks, but were delivered largely as written. The official page for the conference, with video, is here.

Thank you to Matt Lira and Steve Dwyer for the introduction, and to the House of Representatives for holding such an important and timely conference. This kind of event has been a long time in coming.

I must acknowledge the excellent panels that have been happening all day. And I would be remiss if I didn't commend the Committee on House Administration for adopting "standards for the electronic posting of house and committee documents and data," which are already transforming the House in a very positive way.

Because I'm limited to 10 minutes, let me briefly commend three documents to all of you which lay out a transparency vision in greater breath and detail than is possible here. They are the Open House Project Report, the Ten Principles for Opening Up Government Data, and the report from the Congressional Facebook Hackathon.

I've been asked to speak about benchmarks for measuring success in making legislative data available online. I feel like a kid in a candy store, but I will try to restrain myself.  When I speak about the House, please construe my remarks as applying to the Senate and the legislative support agencies as well.

 

What is Transparency For?

In determining benchmarks, it's incumbent on us to assess, at least briefly: what good is online transparency anyway? Here's how I see transparency adding value to our political process. It provides relevant information to decisionmakers at the time they need it. It levels the playing field between the special interests and everyone else so we all have an equal opportunity to find out what's going on. It lets the American people and their elected representatives have a solid basis for a conversation about priorities. It helps congress work more efficiently, by eliminating redundancies and identifying bottlenecks. It allows the agencies to better understand what they're supposed to do. It helps businesses make money by improving their ability to predict government actions. And most importantly, transparency is the cornerstone of a democracy.

This is all pretty ethereal, so I'll get to the point. To the maximum extent possible, legislative information must be available online, in real time, and in machine readable formats. With the exception of internal deliberations protected by the speech or debate clause, or national security and some personnel matters, the Congress's business is the people's business. So let me break down this formulation of online, in real time, and in machine readable formats into concrete benchmarks.

 

Online Publication

Publishing information online is a major hurdle in of itself. A lot of information isn't online, but instead is only available if you know the right person, or go to the right room and ask for a hardcopy, and so on. Should you have to know someone on staff to get a copy of the chairman's mark on a bill before it's voted on? Do we really want to make people trudge down to the House's legislative resource center to print out documents at 10 cents a page? It certainly cannot make any sense to have to request a CRS report through your representative or pay 20 bucks online to buy a copy.

Almost as bad as the failure to publish online is secrecy through obscurity. If information is locked inside an image file and not susceptible to a search engine, or is in an entirely random location, or is hidden on page 400 of the congressional record, it's not really helpful to anyone.

In addition, old information can be just as important as newly created information. For example, there's a huge gap in the availability of committee reports. Along the same lines, while ignorance of the law is no defense for a crime, the actual enactment of the law, known as the Statutes are Large, is not available online for a nearly 80-year period.

Let me offer some concrete benchmarks by which we can judge improvements on this.

1. The House of Representatives should conduct an audit of all the different types of information it produces and releases, including whether it's online, and where it can be found.

2. To the extent the House (or legislative support agencies) has information that is already in electronic format -- from the documents in the Clerk's office to CRS reports to hearing transcripts -- that information should be put online in whatever format its currently in. It's also worth considering whether legislative data should include sometimes released items like Dear Colleagues and Whip notices. We can worry later about improving how this information is made available, but just to start, put them online.

 

Real-time publication

Moving on, let's now talk about real-time publication. This is the kind of idea that makes a lot of people uncomfortable, but I'd suggest a common-sense starting point: think about the time frame and context in which a document is used. An amendment that's going to be voted on in 2 hours needs to be online just as soon as it's drafted. A bill that's going to be voted on in 2 legislative days needs to go up pretty quickly as well. You should know about a committee hearing a week in advance. Other items, like the House disbursement reports, can take a little longer.

Don't get me wrong. The goal should be real-time publication for everything. But the evaluation of what that means in the short term can be context dependent. But that context changes if the document is originally created in digital format -- in that circumstances, there shouldn't be any wait.

Here are some benchmarks:

1. All committee reports, amendments, and bills should be available online as they are introduced. The House should monitor the lag time between introduction and when they appear on THOMAS or the committee websites. I've done this, and it can be a while before some bills show up. Evaluate the extent of the problem, and work to reduce it.

2. All hearing notices should be available online 7 days prior to the hearing.

3. Many committees are skirting House rules about publishing video of hearings. House appropriators are particularly guilty of this. The House should review whether meetings are being held in rooms where video capability exists natively or could be added through use of the House's video service, and pester the committees if they're opting out of recording. When only one meeting in a particular committee is going on at a time, it should be streamed online so long as it is open to the public. It's time to review behavior and start slapping some wrists. Perhaps the House should create a mechanism for the public to report on non-webcast hearings.

 

Machine Readability

So let's move on to discuss machine-readable formats. This is what really allows the idea of House of Representatives as a platform for democracy to succeed.

The biggest wish of many staffers is to be able to dynamically see how an amendment would modify a bill,  how that bill would change the law, (and eventually how an agency would promulgate a regulation, how the courts interpret that regulation, and back to congress again.) Along the same lines, people looking at a bill want to know if there are other, similar bills, in this congress or in previous ones, whether there are committee reports, CRS and GAO evaluations, and so on. If you cannot find a way to tie this information together, this dream becomes impossible.

Legislative data needs to be released as highly structured data. In other words, a machine needs to be able to look at the content and "know" what it is looking at. This would require the use of languages like XML, which allows this kind of value-added context. But to make it work, we also need a way to uniquely describe people and bills and amendments and so on -- cleverly enough embodied in commonly-accepted unique identifiers. There are already tons of these identifiers being used, but the House needs to consistently and widely employ them.

Sometimes, structured language is used when creating a document, or unique identifiers are used to describe data items in a document, but that document is stripped naked before it is released to the public. There are some circumstances where this makes sense, like hiding the different internal drafts of a bill. But most of the time, it serves no real purpose. The data that's removed could be very helpful to those on the outside. Leave it in.

Let me add that PDFs, especially PDFs that are image files, do not promote transparency. They make it difficult to impossible to extract data from documents. If you must use a PDF, make sure that the underlying data is available some other way as well.

That brings me to a point about how the data is made available. A lot of transparency advocates build scrapers to try to transform data that's published online and put it back into a useful structure. Josh Tauburer, for example, scrapes THOMAS to turn it into a database. It's like trying to unscramble an egg.

Legislative data, such as that in THOMAS, should be made available online in bulk. Give folks the database all at once or in very large chunks, and let them figure out how to use it. (See our wiki page for more resources regarding how to improve THOMAS.)

Here are my benchmarks:

1. All bills, amendments, and votes should be published online in XML, or some other structured format. Make scrapers unnecessary.

2. End the tyranny of only publishing in PDFs. House expenditure reports are a giant database -- publish them as a spreadsheet file, not a PDF. The Constitution Annotated is prepared in XML, don't publish it as a PDF.

3. Encourage the use of unique identifiers, whether they come from inside the House or elsewhere. The data needs to be interoperable.

 

Concluding Remarks

My time is running short, so I will only make two more comments about process.

First, today's conference, and the standards released by the House in December, are a good thing.

As a benchmark, we need to have another conference like this one within the next year as a way of assessing how well we have done, and we should continue with these conferences on a regular basis.

Second, we need to foster collaboration between those inside and outside government. In particular, technologists who are trying to use legislative data need to be able to get technology questions answered by the responsible internal stakeholder. And policy works can help provide direction so that the new services developed by the House meet the needs of the public. I suggest:

1. The creation of a standing committee, composed of internal and external stakeholders, that meets at least quarterly, if not monthly, to discuss these issues.

2. A listserv where people who are not in DC can engage in this discussion with people inside and outside of government.

I appreciate your time and the opportunity to speak. Thank you very much.

Yesterday, members of the House of Representatives hosted a ground-breaking public discussion on how to give the public better access to congressional information. Around 300 developers, policy wonks, hill staffers, and others crowded into the Capitol Visitor Center to discuss how to use technology to make the legislative branch more open, transparent, and accessible. The event was sponsored by Majority Leader Eric Cantor and Minority Whip Steny Hoyer.

Matt Lira, the Director of New Media for Rep. Cantor, opened the conference by hailing it as "our television moment," hearkening back to when House proceedings were first televised so they could be watched by the American people. Steve Dwyer, Rep. Hoyer's Director of Online Community and Technology, expressed his hopes that the day's conversation posed "a new model for collaboration between Congressional staff, advocacy groups, and private companies, where we can come together and meet face-to-face over common goals." We could not agree more. Open government is the common ground shared by leaders in both political parties, and we applaud them for their herculean effort to bring people together to work on these issues.

A lot of important information about the ongoing work of the House was publicly revealed at the conference during the first hour, but equally as important, the remaining three hours had attendees break into smaller groups to tackle persistent problems, resulting in incredibly important conversations between staff, technologists, and advocates that rarely occur, and never before on this scale. Intrepid reporter Alex Howard has already published video and photographs from the presentations, and Rep. Cantor posted a short video.

One of the most edifying presentations was made by Reynold Schweickart, the technology guru for the Committee on House Administration, regarding ongoing House efforts to open itself up. Here are the highlights:

• Next week the Committee on House Administration will likely hold a hearing to consider and adopt legislative data standards.

• Along a similar line, the committee is working on improving/implementing legislative drafting in XML, including how to make the data more accessible internally and to outside users. (We can only hope that this includes discussion of bulk access to this information.)

• There are plans to  start publishing floor and committee documents in a machine readable format at permanent URLs. In addition, there will soon be naming conventions for documents that the House rules require to be made publicly available, with the goal of having permanent URLs by 2013.

• GPO, which has begun publishing historic statutes at large online, will start publishing the historic slip laws as individual files, so that you can easily see (and link to) legislation as it was enacted by Congress. (I have a lot more to say about this here.)

• A meeting was held with representatives from all the offices that are involved in creating and disseminating legislative data. If a true collaboration arises, what this could mean is the creation and use of data standards to describe legislation (and its constituent parts) from when it is drafted, through the amendment process, at passage, and upon codification. This would be revolutionary.

• There are ongoing improvements on how video from committee hearings is recorded and made available to the public, with an emphasis on standardizing and making available meta data. (While not a lot of detail was offered, Carl Malamud, who has long advocated for broadcast quality video from the floor and committee hearings, probably has a lot to add on this issue.)

• There's also ongoing efforts with respect to how constituent communications are received by members of congress, and efforts to make it easier to hire capable vendors.

• Finally, there was a stated willingness to consider to what extent the House Rules need to be amended to allow technological modernization that will make the chamber more transparent.

Later on, Darrell Issa, who chairs the Committee on Oversight and Government Reform, announced the launch of "Madison" -- a tool whereby the public can comment on legislation as it is being drafted. Here's a rather grainy photo. Rep. Issa explained the concept: "When a member introduces a bill, it should be interoperably commented on, and [those comments] should be part of the markup consideration. Under the Madison initiative, [interest] group's input will be noted and appreciated, and exposed to the world in real time." While similar in concept to PublicMarkup and Open Congress, the difference is that it would be managed and monitored by the office responsible for reviewing the legislation, giving the opportunity to track ideas (and influence) as it occurs. Indeed, after the conference ended, Rep. Issa's staff hosted a hackathon to help improve the tool so it can be unveiled for public use. Stay tuned.

I haven't even begun to speak about the break-out sessions, which I will briefly summarize. Participants broke into four working groups that focused on the following topics: legislative correspondence, legislative workflow and data, public relations and press relations, and casework and constituent services. We reconvened at the end of the conference to discuss our recommendations for improvements. It's too lengthy to go into here. But, on that topic, I would be remiss to not point to an earlier collaborative effort, the Open House Project, which in 2007 raised many of the same issues and outlined a series of recommendations. (And I can't resist plugging this list of ideas for improving THOMAS).

The outstanding question in my mind is: where do we go from here? Much of the conversation can continue on these open policy and technology listservs, at the hashtag #HackWeTrust, and on pages being set up by Facebook* (who sent many developers to participate in the conference). Even so, it would be great to harness this enthusiasm to hold additional events that bring together experts, staff, technologists, and advocates to address the important but complex questions of how to make the legislative branch open, transparent, and technology-friendly. Similarly, it may make sense to institutionalize this discussion as well, perhaps through working group(s), listservs, or other means.

• Updated to include the Facebook page. Also, check out this colloquy between Reps. Cantor and Hoyer that took place today and discussed yesterday's hackathon.

Want to be the next Director of the Congressional Research Service? This past Monday, the Library of Congress posted a job announcement.

The duties are deceptively simple:

The Director of CRS leads the Service in the research and analysis that supports congressional deliberations on the issues facing the nation. The Director works under the general direction of the Librarian of Congress, reporting on the provision of congressional services and other operational matters, consulting with him on major policy issues, and keeping him informed of significant developments affecting the Service and the Library of Congress.

It's up to the Librarian of Congress to appoint the Director, after consulting with the Joint Committee on the Library. For a taste of the incoming director's challenges, watch this discussion on the future of CRS, hosted by the Advisory Committee on Transparency in May.

A coalition of organizations (including Sunlight) wrote to the Librarian in February to "ask you to appoint a Director of CRS who will help advance the goal of online free public access to CRS reports." It's worth reading the letter in full.

As I outlined earlier this year, it was the last director, and not federal law, that has worked to keep these reports from the public. It's time to restore the public's access to CRS reports, and the best way to start is with a new director committed to meeting the needs of Members of Congress and those who elect them.

Letter to Billington on Public Access to CRS Report

People have become increasingly aware that their elected representatives often do not read the legislation they vote upon. Sometimes there’s not enough time between the introduction and adoption of legislation for anyone but the bill’s sponsor to grasp the contents. Other times so many amendments are added that few people -- if any -- fully understand the final document. In the haste to build support and pass bills, provisions are inserted unnoticed that contain unrecognized flaws and political pay-offs. The public and legislators alike are disconnected from the legislative process intended to serve their needs

Congress and the general public need tools to access and analyze legislation. They need Read the Bill 2.0.

Many politicians and politically-aware citizens seek to reconnect the legislative process through their support of the "read the bill" reform. Its central premise is the idea that legislators and members of the public should have enough time to read legislation before it is voted upon. The Sunlight Foundation, one of the primary advocates of this idea, has long recommended that non-emergency legislation and conference reports be posted on the Internet for 72 hours before final consideration.

In the expiring Congress, this idea gained enough political currency that some of its major legislation -- including health care and financial reform -- was available online 3 days prior to final consideration per  Speaker Pelosi’s promise to do so.  However, no legislative rule imposed this requirement, it was not always imposed uniformly, and often times you had to (1) know where to look to find the bill and (2) be willing to thumb through PDFs containing up to 1,000 pages. Even so, it was a commendable step forward.

Reps. Baird and Culberson introduced legislation (H. Res 554) that would have gone one step further and formalized the “read the bill” requirement by amending the House rules; it garnered 217 co-sponsors. Presumptive-Speaker John Boehner has pledged to impose a “read the bill” requirement for all legislation, but it is not yet clear whether he will update the House rules to do so. We think he should.

Anyone who has tried to read legislation soon realizes that doing so can be complicated and difficult. The text is often written in a shorthand that can be unlocked only with patience and frequent reference to the U.S. Code and Statutes at Large. It's difficult to see how draft language changes over time because there is no redlined version of the legislation, and sometimes you need to compare two lengthy PDFs against one another. The bills themselves don't always show up on THOMAS, the official online legislative database, in a timely manner, and they are difficult to find when published elsewhere.

Congress and the general public need tools to access and analyze legislation. They need Read the Bill 2.0. There are three main technological obstacles.

Legislation should be available online in real time, in machine-readable format, and accessible in bulk.

First, legislation that’s supposed to be available on THOMAS isn’t always there on time. The Senate version of the DISCLOSE Act, for example, took more than 4 days to appear on THOMAS after it was introduced. When legislators bypass THOMAS and directly publish legislation online to satisfy the 72-hour rule, it’s often difficult to find. THOMAS must be comprehensive and timely.

Second, all legislation should be published in a machine-readable format. PDFs aren’t good enough. Unless the data is in a machine-readable format, you cannot easily perform a redline comparison between two bills, like I did here, which showed how the DISCLOSE Act changed over time. Fortunately, 97% of the legislation published on THOMAS this past Congress is available in XML, which is an incredibly useful machine-readable format that provides a helpful structure to the data. But 3% of bills are available in PDF format only, and those bills are usually the ones that are most important, lengthy, and time-sensitive.

Third, the THOMAS database should be available to the public “in bulk,” which would allow users to download large amounts of information at one time. Currently, third parties write programs that extract human-readable information from THOMAS in a time-consuming and error-prone process known as scraping. They then use this information to build citizen-empowering tools like OpenCongress and GovTrack. This kind of innovation -- allowing people to use congressional information to better understand what Congress does -- should be encouraged.

Indeed, in 2009 the Library of Congress and GPO were directed to form a task force to report on the feasibility of bulk access to THOMAS, but no report has materialized. Rep. Foster recently introduced legislation (HR 6289) to partially implement bulk access to THOMAS data and jump-start the report. The incoming Congress should clear the pipes and let public legislative data flow to the public.

Real-time redlining of the US Code become possible once you combine bulk access to information with publishing legislation in a machine-readable structured-data format. Wouldn’t it be great if you can see at a glance how a bill would change the law? Or how an amendment would change the text of a bill? Congress could build this tool, and others, or could require that the legislative text be structured so that outside parties could do so. It could also take steps to improve THOMAS generally. Regardless, nothing will likely happen without congressional action.

The goal isn’t merely to allow legislators and citizens to be able to read the bills, but to understand them as well.

For “read the bill” to have bite, people need better tools to make use of legislative information. Publishing PDFs of bills online 3 days prior to a final vote is necessary, but insufficient, for legislative transparency. The kinds of tools we need can only arise with faster, better access to raw legislative information. We need Read the Bill 2.0. After all, the goal isn’t merely to allow legislators and citizens to be able to read the bills, but to understand them as well.

Yesterday, Rep. Bill Foster introduced a bill that would improve public access to legislative information. Specifically, H.R. 6289 calls for:

• Bulk access to THOMAS legislative summary and status data,
• The creation of an advisory committee that would issue recommendations on improving services provided by THOMAS, and
• The Library of Congress to work towards adding bulk access to the full text of legislation.

This may not seem like a big deal, but it is. In the last year, one in five adult Internet users downloaded or read legislation, according to Pew’s 2010 “Government Online” report. And yet THOMAS, the official public portal to congressional information, has a dated web interface that often obscures its rich contents.

Although there are ongoing efforts to improve THOMAS, its limitations have spurred the private and non-profit sectors to create alternative legislative data interfaces, like GovTrack.us and OpenCongress.org. Unfortunately, their innovative designs are needlessly limited by unreliable access to information.

Public “bulk access” to THOMAS data allows users to download large amounts of information at one time. Technology innovators can mix the reliable information stored in THOMAS with flexible interfaces to address unmet needs, solve new problems, and create more ways for people to make use of legislative data. Currently, third parties write programs that extract human-readable information from THOMAS in a time-consuming and error-prone process known as “scraping.”

In 2009, Congress directed the Library of Congress, the Congressional Research Service, the Government Printing Office, and other appropriate entities “to prepare a report on the feasibility of providing advanced search capabilities” via “more direct methods such as bulk data downloads.” So far, I am unaware of the release of any such report. Rep. Foster’s pushes the matter forward by requiring bulk access to certain types of THOMAS data and calling for all legislative information to ultimately become available for bulk download. (The legislation cited our 2007 Open House Project Report, which called for bulk access to all legislative data.)

The bill takes an additional step by bringing the public further into the conversation about improving THOMAS’s functionality. In the last year or so, the folks who run THOMAS have made serious efforts to get feedback from the public about improving THOMAS, including creating a blog and collecting user feedback. The creation of an advisory committee would formalize the process and ensure that all relevant parties keep talking about how to improve THOMAS. It would also require them to issue regular progress reports.

The legislation is an important step forward and we hope that it receives prompt consideration when Congress returns from recess. In the meantime, the Library of Congress, Government Printing Office, and other relevant parties should act even in Congress's absence. The way forward is clear.

Last year I asked the internet gods for a URL shortener that created permanent links to legislation on THOMAS. Lo and behold, several months later TinyThom.as was revealed, and it is awesome. So once again I cast bread upon the water with three wishes for apps for THOMAS.

1. Compare bills

Bills often undergo a number of transformation before they become law. Understanding the legislative process requires seeing how bill language changes over time – that way you can see when legislators insert unobtrusive but important provisions. When the legislation is made available on THOMAS in XML, it is possible to download the two iterations, paste them into a word processor, and run a text comparison to see what has changed. That's what I did here.

What I would like is an online tool that allows me to avoid all the hard work. It should let me select any two bills on THOMAS and generate a redline document viewable online and available for download. Because some bills are only available in PDF and not XML (or are made available in XML long after the PDF version is online), extra credit would be awarded for anyone who makes it possible to redline PDFs and export the redlined version.

2. Identify related legislation

Sibling bills: In any given congress, multiple versions of identical (or nearly identical) bills are introduced. CRS identifies some of them, but inconsistently, and there is no systematic way to see all bills that contain 97% or 98% + identical language. What would be wonderful is an app that either identifies all related bills in a particular congress, or allows you (when you're looking at a particular bill) to press a button that then lists all other bills that have essentially the same text.

Generational bills: Bonus points would be awarded if you can identify similar bills (or amendments) introduced over multiple congresses.

Kissing Cousins: Mega bonus points would be awarded if you can identify when a section of a bill is identical to another bill (or section of a bill). The same holds true if you can identify when an amendment to a bill is really an existing bill in disguise.

3. Real time redline of U.S. code and legislation

More amendments are introduced for any particular bill than are adopted. What's not always clear is how the amendment will change the text of the bill. This is probably very hard, but what would be great is if there were a way to parse the legislative language to show how a particular amendment would change the language of the bill.

If this isn't hard enough, even more useful would be to see how a bill would modify the US code. (I think this may be impossible with the way bills currently are drafted, which is one reason why the Office of Law Revision Counsel exists, but I would enjoy being surprised.)

In May 2007, Sunlight called on the Library of Congress to "create stable links" to legislative documents published on THOMAS. This recommendation was part of a suite of recommendations to improve that public legislative database, first established in 1995. Although painfully obvious to mention, when hyperlinks are not permanent, people cannot share links to legislation with one another. The old URL dies after a matter of minutes.

The Library of Congress partially addressed these concerns by creating "handles" -- stable hyperlinks to legislative information -- a year and a half later, in October 2008. THOMAS didn't automatically give users these handles. Instead, users had to follow a complicated set of procedures to modify the URL. Few people knew of the feature, or how to take advantage of it.

Not until November 2009 were our permanent hyperlink concerns addressed, by a source outside goverment. Web designer Asa Hopkins created TinyThom.as, which automatically converts THOMAS URLs into stable hyperinks. He also created a handful of useful tools. Although easy to use, not everyone knew that TinyThom.as existed.

It wasn't until this week that the Library of Congress fully addressed our request for permanent hyperlinks. LoC announced a "new toolbar, found near the top of most THOMAS pages, [that] allows users to save or share a permanent link via bookmarks, email, or social networking sites such as Twitter or Facebook. The toolbar also includes quick links to subscribe to THOMAS RSS feeds and to print." LoC has also added a few other bells and whistles, including a top 5 list of frequently searched for bills.

I'm pleased to see that THOMAS is being improved. Its existence is essential to public understanding of congressional activities. THOMAS' user-interface difficulties have spurred the creation of user-friendly legislative resources, such as GovTrack and OpenCongress. However, it's underlying content is one-of-a-kind; it can only be provided by Congress.

That's why we have called for a number of specific improvements to THOMAS. One of the most important is access to THOMAS' data in bulk. What that means is that a member of the public could download all of the information from THOMAS's database at one time, instead of having to access that information through its web interface. Other parts of the government already provide bulk download capability.

It appears that the Law Library (which is part of the LoC) has plans for THOMAS, which they announced in their Holiday letter [PDF]:

For the past two years, the Law Library has assumed greater responsibility for THOMAS (the Library of Congress’ public legislative information system). This year, we launched an analysis of the system’s functionality and content based on user feedback. As a result, we recently implemented changes to the presentation of content and are collaborating with the Government Printing Office to produce a database of all public laws covering 1789 to the present. This content will be added to THOMAS in 2010. Additional legacy content, including congressional hearings, treaties, and floor debates is being prepared for THOMAS in the next few years.

There is an appetite for legislative information, and a community that is hungry to build tools to take advantage of it. Let's hope that these innovations spur others and build a culture of collaboration around THOMAS between those inside and outside the government.