Two Sunlighters recently spent 10 days in Ukraine, meeting with local activists that are fighting corruption and increasing accountability. While there, we learned some valuable lessons to integrate into our own work.
Continue readingClearing up the confusion about our analysis of net neutrality comments to the FCC
We've gotten some intense responses and questions after releasing our latest analysis of public comments on the FCC's proposal to regulate Internet traffic. In this post, we try to clear up the issues that have been raised.
Continue readingOne group dominates the second round of net neutrality comments
A letter-writing campaign that appears to have been organized by a shadowy organization with ties to the Koch Brothers inundated the Federal Communications Commission with missives opposed to net neutrality, an analysis by the Sunlight Foundation reveals.
Continue readingInfluence Analytics: What interests public most at SEC, CFTC? More data disclosure
Proposals to improve public disclosure of corporate and government accountability data attracted the most interest from members of the public, a Sunlight Foundation analysis of 10 years of public comments shows.
Continue readingWelcome to Docket Wrench, SEC and CFTC!
Support from Transparency International and a partnership with Thomson Reuters Foundation has enabled Sunlight to make data from the two important financial regulatory agencies available for analysis.
Continue readingWhat can we learn from 800,000 public comments on the FCC’s net neutrality plan?
FCC Chairman Tom Wheeler’s proposed rules regarding net neutrality resulted in the commission’s largest-ever public comment collection. Use this visualization to browse the hundreds of thousands of documents in a simple and manageable way.
Continue readingFCC releases Open Internet comments in bulk
This afternoon, the FCC announced the release of a bulk, machine-readable archive of the more than 1.1 million comments they received during the first comment period of their Open Internet rulemaking proceeding.
Continue readingAnnouncing the Docket Wrench API
Presenting the new Docket Wrench API! This API allows developers to utilize the same analytics that power Docket Wrench to build innovative new services and tools. Sign up for a free key and check it out!
Continue readingRegulations.gov Continues to Improve, but Still Has Potential for Growth
Recently, the EPA eRulemaking team released a new version of Regulations.gov, a website that tracks the various stages of the rulemaking processes of hundreds of federal agencies, and collects and publishes comments from the public about this rulemaking. We’ve written about Regulations.gov before, and continue to be impressed with the site’s progress in making the sometimes-daunting intricacies of federal regulations more approachable to members of the general public.
This release brings several new features that further this goal. Styling on many document pages has been significantly improved, making it much easier to read both rule and comment text. The presentation of metadata has also been made cleaner, so researchers can more easily find identifiers that help them connect a particular rule to related documents on other websites, such as FederalRegister.gov or RegInfo.gov. New panes have also been added to help users understand the public participation that has occurred so far in a given rulemaking, and to more easily recognize opportunities for further participation.
Of course, since last year’s release of the Regulations.gov API, Regulations.gov is more than just an informational website; it has also become a data provider that now facilitates a variety of third-party participation and analysis tools, as their Developers page now highlights. One such tool is Sunlight’s recently-released Docket Wrench, which uses Regulations.gov data to explore questions of corporate and public influence in the federal regulatory process. Docket Wrench evolved from two years’ worth of effort exploring the possibilities of analysis on federal regulatory comment data, and we believe the time we’ve spent building it has given us a unique perspective on the avenues of research this data makes available, as well as the opportunities for further growth and improvement in regulatory comment data going forward.
The team behind Regulations.gov deserves enormous credit for the progress they’ve made, but there remains much work to be done to give the public a complete, accessible and useful path into the federal regulatory process.
Continue readingRegulations.gov Gets an API & More
Sunlight has been interested in the federal rule-making process for quite a while: we sponsored the app contest that lead to the current incarnation of federalregister.gov, which lists federal regulations as they are published, and kick-started an effort to map regulations to the laws that authorize them during a hackathon late last year. We also have extensive experience in the analysis of corporate influence on the political process, having launched several prominent influence-related projects under the Influence Explorer banner. During the last year, we’ve begun to examine the confluence of these two interest areas: corporate influence on the regulatory process, and, in particular, the comments individuals and corporations can file with federal agencies about proposed federal regulations. The first glimpses of the results of this effort went live on Influence Explorer last fall, with the addition of regulatory comment summaries to corporations’ profile pages.
Given this history, we’ve been excited to explore this week’s relaunch of regulations.gov, the federal government’s primary repository of regulatory comments, and the source of the data that powers our aforementioned Influence Explorer regulatory content. This new release brings with it a much-needed visual spruce-up, as well as improved navigation and documentation to help new users find and follow regulatory content, and a suite of social media offerings that have the potential to expose rule-making to new audiences. There have also been some improvements to document metadata, such as the addition of category information visitors can use to filter searches by industry, or browse rule-makings topically from the homepage.
Of more interest to us as web developers is the addition, for the first time, of official APIs to allow programmatic access to regulatory data. It’s clear that the regulations.gov team has taken note of current best practices with respect to open data APIs, and have produced clean, RESTful endpoints that allow straightforward access to what is, especially for a first release, a reasonably comprehensive subset of the data made available through the general end-user web interface. While we have been successful in performing significant regulatory analysis absent these tools, our work required substantial effort in screen-scraping and reverse engineering, and we expect that other organizations hoping to engage in regulatory comment analysis will now be able to do so without the level of technical investment we’ve had to make.
Of course, there is still work to be done. Much of the work we’ve done so far on regulations, and that we hope still to do, revolves around analysis of the actual text of the comments posted to regulations.gov (which can take the form of PDFs and other not-easily-machine-readable documents), and depends on being able to aggregate results over the entirety of the data, or at least significant subsets of it. As a result, even with these new APIs, we’ll still need to make large numbers of requests to identify new documents, enumerate all of the downloadable attachments for each one, download these attachments one at a time, and maintain all of the machinery necessary to do our own extraction of text from them. While we’re fortunate to have the resources to do this ourselves, and have made headway in making the fruits of our labors available for the public, it would certainly behoove the regulations.gov team to move forward with bulk data offerings of their own. Sunlight has a long history of advocating the release of bulk data in addition to (and perhaps even before) APIs, and the regulatory field illustrates many of our typical arguments for that position; the kinds of questions that can be answered with all of the data are fundamentally different than those that can be answered with any individual piece. We recognize that offering all of the PDFs, Word documents, etc., to the public might be cost-prohibitive from a bandwidth point of view, but regulations.gov is doing text extraction of their own (it powers the full-text search capabilities that the site provides), and offering bulk access to the extracted text as we have done could provide a happy medium that would facilitate many applications and analyses without breaking the bandwidth bank.
In general, we see plenty of reasons to applaud this release and the team at EPA that's behind it. While many of its changes are cosmetic and additional improvements will be necessary for regulations.gov to reach its full potential, this update promises further progress that will benefit developers and members of the public alike. We share the enthusiasm of the regulations.gov team for increasing access to and awareness of these crucial artifacts of the democratic process, and look forward to engaging with them and the broader open government community as they continue to improve this public resource.
Continue reading
59063
