A few weeks ago, we integrated the full text of federal bills and regulations into our alert system, Scout. Now, if you visit CISPA or a fascinating cotton rule, you'll see the original document - nicely formatted, but also well-integrated into Scout's layout. There are a lot of good reasons to integrate the text this way: we want you to see why we alerted you to a document without having to jump off-site, and without clunky iframes.

As importantly, we wanted to do this in a way that would be easily reusable by other projects and people. So we built a tool called us-documents that makes it possible for anyone to do this with federal bills and regulations. It's available as a Ruby gem, and comes with a command line tool so that you can use it with Python, Node, or any other language. It lives inside the unitedstates project at unitedstates/documents, and is entirely public domain.

What it Does

The chief problem that us-documents solves is taking the original documents and converting them into context-less HTML that can be dropped directly into any website. Once dropped in, all of the styling can be done with CSS.

For example: Congress publishes XML for every bill in the House and Senate, and it's rich XML (just look at that DTD). It's built to handle a lot of different use cases, including compatibility with sophisticated drafting tools. Most of that just gets in the way of displaying the bill to users, so we can rip most data out and turn it into div's and span's.

In this before-and-after example, we're turning a piece of CISPA's official XML into HTML we can drop into place:

Doing it this way also lets us make our own decisions on what to display - Congress may feel the need to display the table of contents of the bill, but we don't, so it can be hidden with CSS.

We do something similar with rules and notices from FederalRegister.gov. Even though FR.gov already provides HTML snippets ripe for integration, there are simple things we can do to make them even more universally usable, like ditching "id" attributes.

This lets us take the HTML used here on FederalRegister.gov and embed it plainly in Scout, without any conflict with our own HTML.

Doing Right By Citations

One of the other reasons to integrate these documents into Scout was to link legal citations to searches, to take advantage of Scout's special citation searching.

Both Congress and the Federal Register already attempt to detect and link the legal citations in their documents. To make these links easily overrideable, us-documents extracts the basic pieces from each caught citation and puts them into data attributes on the link, e.g. data-title="5" data-section="552".

This way, we can easily process that HTML client-side in JavaScript, and replace the original links with new ones built from those data attributes.

The Federal Register does a terrific job of detecting citations, but strangely, Congress's detection seems a lot spottier. If you take a look at last year's DISCLOSE Act, it's difficult to tell why only one citation gets linked, and so many others are not. It's possible we may end up using our own citation detector in the end.

Does This Need a Standard?

us-documents is doing a pretty naive transform on these documents, and the resulting HTML needs entirely different CSS for bills and for regulations. It works, but there are few questions that come to mind:

• Is it worth identifying the common denominator of tags and features necessary for both kinds of documents, and transforming them both into a standard end product?
• If so, should that standard be the resulting HTML, or is it worth creating an intermediate format of some kind?
• And should this standard have some sort of compatibility or at least good vibes with existing standards for legal documents, such as Akoma Ntoso or legal-markdown?

The answers may become more obvious over time if others begin using the tool, and if it gets expanded to process more kinds of documents than just federal bills and regulations. Grant Vergottini also just wrote a post asking smart questions about the use of HTML, XML, and JSON with documents like these.

In the mean time, us-documents has already been useful to us, and we hope it will be to others.

Next Friday will be my last day as Sunlight Foundation's policy counsel. On June 3rd, I join CREW as policy director. It's been an amazing four years.

I joined Sunlight when it was less than half its current age and had fewer than half as many staff. At the time, I was the second member of the policy team, and our focus was legislative branch transparency. Since then, Sunlight's advocacy has expanded to the federal executive branch, municipal governments, and foreign nations.

All the while, we've brought our vision of technology-facilitated transparency to a global audience, and transformed a technology-focused nonprofit with a policy sideline to a policy-focused nonprofit that uses technology to change the world.

I've had a front seat to that world-wide transformation. We've cheered as the House of Representatives embraced the public's right to unfettered access to legislative information. We've applauded the White House's Open Government Directive and Open Data initiative. We've encouraged the formation of a 120-member 80-nation network of parliamentary monitoring organizations. And we've been involved as important legislation, like the DATA Act, ACMRA, the DISCLOSE Act, CRS Electronic Accessibility Resolution, POIA, Lobbying Disclosure Enhancement Act, and many others, as they have worked their way through the political system.

Our work isn't done. Our work is far from done.

My new employer, Citizens for Responsibility and Ethics in Washington (CREW), uses different tools to promote transparency, ethics, and accountability in government, even as much of the mission overlaps. I'm looking forward to working with Sunlight and all members of the transparency community to continue the good fight.

I cannot sufficiently express my gratitude to Ellen Miller, John Wonderlich, and everyone at Sunlight (past and present) for making my time here so rewarding. It has been a pleasure. I'll see you on the Hill.

This Wednesday, the House of Representatives is holding its second annual Legislative Data and Transparency Conference. (RSVP here.) The conference provides an opportunity for legislative staff, policy advocates, and the general public to have a wide ranging conversation about how congressional information is made available to the public, and how public access can be improved.

The conference represents a real effort by the House of Representatives to engage the public with its efforts to become a more open and transparent institution.

The conference will take place in the Capitol Visitor Center Auditorium from 9-5. That evening, a number of organizations, including the Sunlight Foundation, will host a happy hour from 5:30-7:30. (RSVP here.)

The conference's agenda is now online. It is reproduced below:

8:30 -- Registration

9:00 -- Conference Welcome

9:15 -- Legislative Process Overview: Bill Preparation Process and Incorporation into U.S. Code

9:45 -- Legislative Branch Update: Bulk Data, Electronic Access to Legislative Information, LOC & GPO Digitization, Technical Developments

10:45 -- Morning Networking Break

11:15 -- Official Tools Demo: Operation of Committee Repository (docs.house.gov), Searchable Committee Roll Call Votes, Party Intranet

12:15 -- Lunch Break

1:30 -- International Update: 2012 XML Usage Data, International Data Standards, 21st Century Innovations

2:00 -- Electronic Legislative Archiving: Lessons on Archiving, Capabilities of GPO's FDsys, Strategies for Permanent Public Access

2:40 -- Afternoon Networking Break

3:10 -- Improving Access to XML: Extending and Normalizing XML

3:50 -- Under-Digitized Legislative Data: Increasing Publication of Digital Legislative Data to Maximize Usability

4:30 -- Conference Wrap Up