When we talk about providing a transparent and legible legislative process, the first step is identifying all the moving parts that contribute to the passing of law. In Washington, D.C., this means not just following the city council, but also following D.C.’s Advisory Neighborhood Commissions, or ANCs, the hyperlocal government entities that advise the city council on community issues. Although the District’s city government structure is uniquely granular, it serves as an excellent model for understanding what a complete legislative data framework should consist of.
By Carrie Tian and Matt Rumsey. Research Assistance by Justin Lin.
Every morning in Washington staffers, lobbyists, activists, and ordinary citizens are faced with choices as they try to schedule their days on Capitol Hill. To fill their calendars and get to hearings on time they have to navigate several, often conflicting, sources of information to find the right date, time, and hearing room. As a result they can find themselves checking their laptops before leaving the house and refreshing their phones as they rumble down the Redline.
Docs.House.Gov, among its other features, aims to simplify this problem by becoming a one-stop repository for information on House committee hearings. We decided to look back at the first six months of this program to see how close it was coming to the ideal of including every House committee hearing, as compared to the other sources of “the same” information, notably house.gov/legislative and individual committee websites.
This map distinguishes five levels of legislative web and broadcasting comprehension on a sliding scale from “Best” (including all recommended elements: video formatting of floor proceedings and committee hearings, archived, and broadcasted via a variety of mediums) to “Worst” (missing several of these recommended elements). For more info (or to watch!) see the NCSL's original roundup here.
Open legislative data is integral to a functioning legible participatory democracy. The legislative data canopy covers everything from information about who represents you to the nuts and bolts of the legislative process to final letter of the law, with each element carrying its own series of challenges and considerations when it comes to public access. Timely and archived legislative process data (i.e. bills, amendments, committee meetings, votes, and contextual information, such as: research reports, legislative journals and lobbying information) are crucial to supporting citizen participation and informed voting. Video documentation of the legislative process represents the barebones of open and accountable legislative process data -- passive recordings of events as they happen for prosperity and public inclusion -- and yet this information is still not comprehensively available in most U.S. states.
by Daniel Schuman and Eric Mill
House Appropriators may deal a tremendous blow to prospects for improving public access to legislative information. In a draft report expected to accompany the Legislative Branch Appropriations Bill for 2013, scheduled for a full committee vote tomorrow, appropriators misunderstand how data can be "authenticated," and kick responsibility for improving public access to legislative data to a non-public task force with no set reporting date. Unless corrected, this draft report represents a tremendous step backward for transparency, and fails to seriously grapple with the history of efforts to free legislative information for widespread public use.
Legislative Information Should Be Widely Disseminated
The purpose of THOMAS is to bring legislative information to as many people as possible; preservation and authentication is best handled through other long-established methods that THOMAS was never intended to address. The lack of authenticity to THOMAS data does not present a problem for most users. Rather, the largest problem with THOMAS is that the data is not provided so that it can be easily copied, placing a significant burden on citizens who wish to make sophisticated use of the information. The THOMAS website directly provides nearly a million users each month with an "inauthentic" version of information about legislative activities, a practice that will continue unabated under the draft committee report. While THOMAS often links to a GPO document that is "authenticated," its display of bill text, legislative summaries, cosponsor data, and other information is not certified as being correct, and often changes because of the Library's errors in how it publishes the data.
To the extent to which THOMAS information should be authentic, the report does not engage with best practices around authenticity of data on the Internet. Verifying the authenticity of data can be performed securely and reliably with the use of metadata external to the data itself. In fact, this is precisely how GPO's FDSys currently authenticates XML documents of the US government, including its legislation, regulations, and laws. GPO accompanies each document it publishes with a "PREMIS" metadata file that includes information needed to cryptographically verify the authenticity of documents. For example, here's the PREMIS file accompanying HR 6289. Worries about authenticity are a red herring.
Bulk Access is a Separate Question from Authenticity
Bulk access to THOMAS data is a simpler and less controversial step than this draft report contemplates. The underlying information is already publicly available on the THOMAS website, and third parties already are scraping the data from the site to make it available in bulk. It simply makes sense for the Library to meet the needs of the public directly through providing the data in bulk itself. This merely opens up another avenue to access info that's already being released. It would also eliminate any errors created through the scraping process.
The Draft Report Creates a Secret, Never-Ending Process
The draft report would require the establishment of a task force to examine and report back on a number of issues raised in the report regarding bulk access to legislative data. This is seriously flawed in several major ways.
First, bulk access is about granting the public better access to legislative information. It stands to reason that the public should be included in all discussions. However, the proposed task force does not include any non-governmental participants. A number of individuals and organizations are expert in these matters, and should be full participants.
Second, the draft report imposes no deadline for a report from the task force. The last time Appropriators required a task force on a similar matter, four year ago, it never reached any conclusions or reported back. Without a deadline, the same will happen here.
Third, the task force's report should be provided to the public as well as to the committee at the time it is completed. Draft reports should also be made available for public comment.
Fourth, the report language is terribly overbroad: it prohibits the establishment of bulk data downloads of legislative information prior to the reporting back of the task force. Making use of modern technology to provide information in better ways should be something that is encouraged, not prohibited. Information is already being provided to the public in bulk regarding certain legislative activity. Would this report language stop the GPO from providing bulk access to the Congressional Record, as it does now? Would it prohibit the House of Representatives from providing bulk access through its innovative docs.house.gov portal? If so, that would be a disaster for transparency.
Finally, the idea of a task force to assess these questions ignores that these issues were already addressed by the Library of Congress in a 2008 memo.The memo explained that the XML database containing bill metadata was expected to be able to be released in bulk by May 2008. It also stated that "CRS... will continue to identify and analyze ... the following policy matters for the Committee's consideration," including "data accuracy" and "data permanence and authentication." Where are the results of CRS's analysis? What is the strategic plan for THOMAS referenced in the memo? Where is the study promised that would engage in "an examination of permanence and authentication of legislative data, along with any attendant issues, risks and workload?"
Simply put, the draft committee report's establishment of a task force is another recipe for delay. We saw this four years ago, the last time the Library was pressed to make improvements on this issue. The time is long past for action, and the Appropriations Committee will be judged on whether it makes another plan to make a plan, or whether it establishes real deadlines for progress. THOMAS itself was created in a matter of months when the Speaker of the House decided it was a priority. Bulk access to legislative data will also come about when legislators decide that being transparent is more important than establishing a task force to talk about it.
Today, the House of Representatives announced it will host a full-day conference on public access to legislative information on Thursday, February 2. This is a big deal. It will bring together the people who create and encode legislative materials and the people that use (and transform) that information. This announcement follows on the recent launch of a House transparency portal, which in of itself will change how the public makes use of legislative information.
Entitled "Legislative Data and Transparency," the conference will include discussion of how legislative information is created, how it is made available to the public, what the impact is of current levels of public access, what improved public access would look like from a technological perspective, and the benchmarks to determine and benefits that would come from a truly transparency Congress.
In May 2007, the Sunlight Foundation gathered a coalition of organizations to make recommendations on what an open House of Representatives should look like. While some of the recommendations have been implemented, five years on there's still a lot to do. Deepening communication between those on the "inside" and "outside" will only help to make Congress a more responsive, efficient, and transparent institution.
This conference has been a long time in the making, and I congratulate the often unsung congressional staff who have labored long hours to make it happen, as well as the political leaders who have demonstrated the determination to make this happen. I also must disclose that I've provided advice about what I think the conference should look like.
I hope this serves as a kick-off to many more discussion between those inside and outside Congress about how to fully bring it into the Internet age. RSVP here.
At a Friday hearing, the House of Representatives significantly raised the bar on open data by passing a resolution requiring that a wide variety of crucial House legislative information be published online, in open formats, and at permanent predictable URLs. Daniel Schuman covered this on the Sunlight Foundation blog on Friday.
The new standards create a new central website, run by the Clerk of the House, that will host all House bills, resolutions, amendments, and conference reports. These documents will be online on January 1, 2012, and will be in XML.
Beyond that, the standards require committees to post their amendments, votes, hearing notices, which bills and resolutions they're considering, and lots of other documents. The Clerk is charged with building tools for committees to post this information to the new website; in the meantime, committees must post them to their own website, in PDF. Committees are also encouraged to post this information in XML, and "should expect XML formats to become mandatory in the future".
This is hugely valuable information that, to date, has been extremely difficult to discover in a reliable way. To get House legislation, one either needs to scrape THOMAS.gov (a Sisyphean ordeal), or to rely on the good work of people who've already done it. Committee information is terribly fragmented, and in some cases there is often no way to get it at all (such as committee votes and amendments), short of hiring people to go sit in committee rooms and record what goes on (a practice that forms the basis for a number of business models here in DC). This is the beginning of bringing much needed order to chaos, and sunlight to the legislative process.
These standards demonstrate excellent leadership on the part of the House, and offers a modern vision for how a legislative body should view its responsibilities to the public. The Senate should hear the sound of a gauntlet being thrown. The Committee's action is in keeping with Speaker Boehner's and Majority Leader Cantor's April call for the House Clerk to release legislative data in machine readable formats. It is very gratifying to see this call taken so seriously.
This morning, the House of Representatives took a tremendous step into the 21st century when the Committee on House Administration unanimously adopted "Standards for the Electronic Posting of House and Committee Documents & Data."
Taking effect on January 1, 2012, the resolution instructs the Clerk of the House to maintain a single website where the public can access all House bills, amendments, and resolutions for floor consideration in XML. In addition, committees will be encouraged to post their documents on that site in XML whenever possible -- and searchable PDFs when not -- with the expectation that mandatory publication requirements in XML will soon be imposed. The House will also store video of hearings and markups, and work to implement standards "that require documents to be electronically published in open data formats that are machine readable," thereby enabling transparency and public review.
In a statement, Committee on House Administration Chairman Dan Lungren said “With the adoption of these standards, for the first time, all House bills, resolutions and legislative documents will be available in XML in one centralized location. Providing easy access to legislative information increases constituent feedback and ultimately improves the legislative process. ”
Three cheers to Chairman Dan Lungren, Ranking Member Bob Brady, members of the committee, and its staff for moving this important issue forward. As was discussed at the recent #hackthehouse conference, as well as in our longstanding Open House Project Report (pdf), there's a lot more to do, but this is a major stride towards implementing Speaker Boehner and Majority Leader Cantor's pledge to " publicly releasing the House’s legislative data in machine-readable formats." The Senate could do well by following this example, as could legislative support agencies like the Library of Congress and GPO.
This is a continuation of Statelight, our series on advocating for open government change on a local level. Depending on which issue you chose to focus your advocacy on, you will need to customize your outreach. To help you do that, we’ll be following up this post with a series of short guides to help you organize around the convoluted terrain of state politics, but first, we’ll lay our some general organizing principles that you’ll need to know no matter which issue that you choose.
I. Pick your Issue
It may sound basic, but since transparency encompasses an enormous set of issues, it’s important to focus your advocacy on one in particular that's relevant to your interests or state. As we outlined before, we’re focusing this guide on three winnable open government issues: ethics and campaign finance, budget transparency, and legislative data. You can learn more about researching these subjects here. Remember: You don’t have to become an expert in your issue, but you do need to learn how to ask the right questions.
II. Register your project
Once you’ve decided on an issue, it’s time to work towards change. Register your initiative with us at http://publicequalsonline.com/projects/ so that others can find and connect with you. Don’t worry about how far along you are or whether or not you have all the kinks worked out..You can keep coming back to update your project as you go along. For the short term, it’s just important to get the information out in public so that people with similar interests to you can connect with the work that you’re doing. Not only will this help you grow your base of volunteers, but once you make register and make your project public you’ll also be able to connect with folks across state lines to discuss best practices for the policy you’re focusing on.
III. Build Your Team
Part One: Targeted Outreach
The most effective advocacy demonstrates a need for change across constituencies. So, target your outreach. At this point, you know the policy you’re focusing on, who the governmental players are that have the power to change that policy, and what you want that policy to become. Now it’s time to build your team. Ask yourself, what groups have a stake in this policy? Who are the groups that regularly meet with the government officials that make this policy? These questions should help inform who you need to recruit for your team.
Although you’ll want to specialize your team to some degree depending on the issue you’re organizing around*, here are some general recruiting tips to follow as you look for volunteers:
An immediate way to reach out to people willing to work with you for meaningful change is to reach out to the community of open government folks already working in your area. Check out this list of open government and transparency organizations working in your state. Even if you find that the organizations working in your state are not particularly focused on your issue, it’s still worth contacting them as they may have resources that could be useful to you and people willing to support you in your endeavor.
Two other groups to look for are state/local watchdog bloggers and city-level open government activists. Sunshine bloggers will be useful in your future media outreach and often have knowledge about particular challenges to working with data, policymakers, and other relevant issues that will help you. On the other hand, folks invested in open cities are likely interested in state-level transparency, too, and since your work may impact theirs it’s important to look for opportunities to support each others work.
Having trouble connecting with folks or thinking of more people to talk to? Don't be afraid to think creatively about groups to contact in your state. Transparency issues impact a number of fields, including healthcare, the environment, and civil liberties. Consider touching base about your work with labor unions, campus groups, land trusts, libraries, neighborhood associations, non-profits, and sports clubs who have expressed an interest in issues overlapping yours. Like-minded groups will not only present you with the opportunity to enlist volunteers, but will also create contacts for you to spread the word to about the work that you're doing.
Part 2: Make the Ask
Once you’ve established who you need to reach out to, make the ask.
Explain to the organizations and activists that you’re reaching out to what your goal is and ask them to join you. Your coalition can be official or unofficial; the most important thing is that you set up a meeting, conference call, or listserv where you clearly lay out your issue, what you want to change, and what your agenda is for doing so.
Once you’ve set this up, check back to your project page to update this information on your project page for others who may want to get involved. This is an easy way to make sure you document your progress and will help keep you from losing potential volunteers who can spread the word about your development.
In our next post, we’ll dive deeper into issue-specific organizing. Stay tuned for How to Organize for Open Legislative Data!
*Don’t worry: as we continue with our installments covering issue-specific organizing, we’ll include a reference for your policy’s Advocacy Dream Team. Stick with us.
As we set the stage for advocating on local and state transparency issues and highlight some policies issues of note, inevitably the question comes:
How exactly do you go about learning more about policies in your state?
Whether you've already picked a policy focus, want to learn more about what role open government policies could play in your state, or are just chasing curiosity, you need to answer this question. To make your search for answers clearer -- more transparent if you will -- we've laid out a series of questions for you to ask to help get you started with your research.
ONE: What issue area do I focus on?
In our last episode of Transparency in a Box, John laid out three issue areas that will be of interest to many state advocates: ethics and campaign finance, budget transparency, and legislative data. Depending upon which policy you choose there are different resources that outline the answers to a crucial question for your research:
What information does the government currently provide about this policy? Or, what's knowable from your government?
The answer to this question will vary by state and issue. If you're interested in learning about how to determine what the situation is in your state for ethics and campaign finance, jump down to TWO. If you're looking to do research on budget transparency, jump to THREE. And for your legislative data needs, head to FOUR. For general open government resources, jump to FIVE.
TWO: How can I figure out what information is available for ethics and campaign finance?
Where to look: Your state likely has a department of government that oversees campaign finance laws, either as part of the office of its Secretary of State or Board of Elections. (For a good example, check out Michigan's site for Lobbying Disclosure.) Personal Financial Disclosures can be harder to find and are sometimes available from a statewide ethics office. (See Florida’s example. ) For a beginning look at your state's code of law, the Library of Congress provides some great links, though you may need to do an additional search to find out your state’s ethics regulations. And while not a perfect resource, you should also play with CREW's Ethics in Your State tool. Although the corruption rankings are two years old, you'll find helpful resources for groups working on ethics and campaign finance in your state along with media resources that might help give you a greater sense of your state's situation.
What to look for: You can focus on one of the three policies below or search for information on all of them. In each instance, you'll want to find out whether these things are actually available and if so, how much information is provided to the public and how much is online:
- Personal Financial Disclosures
- Campaign Contributions
- Lobbying Disclosure
How this policy is enacted: Campaign finance policies are determined by the passage of laws or by executive orders. Finding this sort of information will do a lot to tell you about why things are the way they are now, which is important to understand if you're interested in working to make this policy more transparent.
THREE: How can I figure out the transparency of my state's budget?
Where to look: Your state's budget portal. If you're unable to find what you're looking for there, try looking for more information on your governor or comptroller general's website. Although you should always go the source (i.e. your state's official website) there are other sites that can help inform your search and understanding. For instance, check out this resource from the Center for Fiscal Accountability, which hosts useful links and news about budget transparency, listed by state. The Sunshine Review also posts a guide to budget transparency listed by state.
What to look for: Are budget proposals posted online in addition to actual expenditures? How timely or complete is the publishing? Does your state have an open website for its budget/spending?
How this policy is enacted: Many states (but not all) have an online budget portal that shows how money is spent. These portals could be enacted by law, but are more likely based on an initiative from your governor or your comptroller general who can build such a site when made a gubernatorial priority.
Related to this portal is the general process by which your state sets its spending priorities. This system can vary from one state to another, but generally involves your governor submitting a proposal for a budget and your legislature eventually approving it. The actual process, from submission to approval, is incredibly complex with a number of open and secret negotiations.
FOUR: How can I figure out what legislative data my state publishes?
Where to look: The official source of legislative information in your state, most likely your state legislature's website, which you can find quickly on this comprehensive list. (Thanks, Wikipedia!) The Library of Congress also publishes a resource on state laws that should help your search. Each state page includes links for that state's bills and legislative session laws. The Open States Project is also working on collecting a variety of legislative data for each state and may be a good source for research or collaboration.
What to look for: Keeping in mind that there are variety of ways to interpret "legislative data transparency" (which John goes into more detail with in his post), two to focus on:
Look to see if your state legislature posts its bills, amendments, and votes online. See whether the information is searchable and what the timeline is that your state uses for posting these documents online. Does your legislature note whether bills have to be online before a vote? If so, how long before?
You should also check to see how the state is making its legislative data available. This is the more technical side, but it’s crucial to creating innovative web platforms that can aggregate and share bills, votes, and amendments. Most states display bills and other legislation in HTML; others post actual data files in order to empower more advanced analysis and reuse. For a great example of open government work in this field, see the New York State Senate, a leader in making raw data available to developers.
If you're having problems searching for information on this subject on your legislature's website, you may have to turn to your search engine of choice. Use a combination of keywords including your state's name and phrases like "legislative data," "general assembly data," "data system," and "online." Your search results may turn up a free resource (see Rhode Island's example) or a proprietary one. Look for contact information from an associated government official to learn more.
How this policy is enacted: Legislative data is controlled by legislative procedure and changes to existing policy may require a change of your assembly or chamber's rules, or may also be able to be changed by an initiative of the technical staff of the legislature.
FIVE: Where can I learn more about general states-related open government stuff?
There are number of resources scattered around out there, but a good place to start is to check out the work of organizations within your state. This directory of state and national organizations is a great place to find out about who's already working on issues in your state and what issues they focus on. My colleague, Sarah, is also compiling a review of state transparency efforts. Check back to see how your state is doing.
While not an open government site, per say, this digest of state laws, hosted by the Library of Congress (mentioned above) is a great "one-stop" source for state-specific government sites. Head here when you're looking for pages listing your state's bills, legislative session laws, and other odds and ends.
OpenMuni Wiki is a collaborative platform where people share case studies and best practices for open government policy and advocacy. Although the site started with a more local focus, it has since expanded to include state-level concerns and should be turned especially when you're looking into open standards for government data. If you're looking for even more ideas or inspiration from the work others are doing around the country, OpenMuni offers a great list of like minded efforts. Poking around these groups will show you the lay of the open government land and again provides the opportunity to collaborate on ongoing transparency projects. Similarly, check out CityCamp to see how folks working on a municipal level are actively gathering public officials, citizens, and experts of every sort together to problem solve local transparency issues. Getting involved with a CityCamp can be a great way to connect with people already in the field.
This post only gives you an tiny view of all the work being done to get transparency for different levels of governance. There is still a great need to organize for change on a state level, and to be successful these transparency initiatives need to communicate with one another. If you do decide to join an open government group or to craft your own initiative, post your efforts where others can find and connect with you and your work. We host an open government projects page so that you can archive your work, ask questions of your peers in the field, and check out the work others are doing, though connecting with folks over Govloop or OpenMuni will be useful to you, too.
Have a favorite open government site that I missed? Please (!) share it with us below. (Extra points if it has a state focus.)