Linting your code reveals syntactical errors and deviations from accepted stylistic conventions, and — as someone that switches between languages a lot — I heartily recommend it.
Continue readingHow to quickly set up Travis CI for your Ruby application
If you want your Travis build to go faster and you use [`bundler`](http://bundler.io/) in your Ruby project, try adding this to your .travis.yml.
Continue readingRecap: RailsConf 2011
Last week at RailsConf, Sunlight held an Open Government Hackathon at the event's official unconference, BohConf. We worked on a few projects:
- The Ruby gem for the Sunlight Congress API was updated. We added support for new fields on the Legislator resource and removed support for deprecated features. The developers behind search.usa.gov (a Rails app!) were there, and led development on a Ruby gem that wraps the site's product recalls API. Dan Melton from Code for America led development on a Ruby gem that wraps the Small Business Administration's API.
Speaking of Dan, he gave a keynote at RailsConf the next day:
Thanks to all who made it out and contributed!
Continue readingPreview: Real Time Congress API
My main project for the last month or so has been something we're calling the Real Time Congress API. It's not quite ready for production use, and the data in it is subject to change, but I wanted to give you all a preview of what's coming, and to ask for your help and ideas.
The goal of the Real Time Congress (RTC) API is to provide a current, RESTful API over all the artifacts of Congress, updated in as close to real time as possible. For the first version, we plan to include data about bills, votes, legislative and policy documents, committee schedules, updates from the House and Senate floor, and accompanying floor video.
Continue readingScraperWiki is Extremely Cool
ScraperWiki is a project that's been on my radar for a while. Last week Aine McGuire and Richard Pope, two of the people behind the project, happened to be in town, and were nice enough to drop by Sunlight's offices to talk about what they've been up to.
Let's start with the basics: remedial screen scraping 101. "Screen scraping" refers to any technique for getting data off the web and into a well-structured format. There's lots of information on web pages that isn't available as a non-HTML download. Making this information useful typically involves writing a script to process one or more HTML files, then spit out a database of some kind.
It's not particularly glamorous work. People who know how to make nice web pages typically know how to properly release their data. Those who don't tend to leave behind a mess of bad HTML. As a result, screen scrapers often contain less-than-lovely code. Pulling data often involves doing unsexy thing like treating presentation information as though it had semantic value, or hard-coding kludges ("# ignore the second span... just because"). Scraper code is often ugly by necessity, and almost always of deliberately limited use. It consequently doesn't get shared very often -- having the open-sourced code languish sadly in someone's Github account is normally the best you can hope for.
The ScraperWiki folks realized that the situation could be improved. A collaborative approach can help avoid repetition of work. And since scrapers often malfunction when changes are made to the web pages they examine, making a scraper editable by others might lead to scrapers that spend less time broken.
Continue readingSunlight Hackathon at RubyConf
I'm happy to announce that Sunlight will be hosting an open government hackathon at RubyConf. The three day event will be held in New Orleans from November 11 to 13. This is the big show in the Ruby world, where the Ruby core team from Japan will make the trip across the Pacific. While the event initially sold in a matter of days when registration first opened, a second batch of tickets will be made available today. So head on over to the RubyConf registration page right now to get your seat.
The RubyConf organizers, Ruby Central Inc., have been gracious enough to provide us with a dedicated room at the conference venue. Previously, Sunlight has hosted successful hackathons at PyCon around the Open States Project. With the hackathon at RubyConf, we plan to have a diverse selection of projects for developers to work on. We'll have projects at the federal level and state level, like writing data importers for the National Data Catalog. In the past year within our community, there's been tremendous energy at the local level. We encourage those working on city-centric efforts to bring "shovel-ready" projects to the hackathon, and we'll support you throughout RubyConf. Just get in touch with me and we'll get your project on the board. We're also working with key stakeholders in New Orleans, and hope to have some projects ready that can directly help the city in its recovery efforts.
Continue readingThe National Data Catalog Is Hungry
So you've found some government data on the web. Naturally, you are eager to share your findings with the world. Perfect! Sunlight Labs can help. Our National Data Catalog (NatDatCat) is hungry for government data, and we have to feed it regularly. Otherwise, it gets grumpy.
The first step is to assess what you've found. If it is just a few bits of scattered files, just fill out a quick form and tell us about it. On the other hand, if it is a collection of data sets, you might consider writing an importer...
Continue readingThe Drumbone API
On our new API homepage, we recently added the Drumbone API. It's a light, flexible, JSON-only API over Congressional legislator, bill, and vote data, and we currently use it in two of our products. I wanted to take a minute and explain why we built this, especially in the face of the existing suite of community sources for this data.
Continue readingRecapping the Ruby Hoedown
I was in Nashville this past weekend giving a talk at the Ruby Hoedown, a completely free conference held for the South's Ruby developers. My talk was titled "Civic Hacking", and the slides are on Slideshare.
Continue readingIntroducing Congrelate
I've been tweeting about this for quite some time, but I should announce it on the blog too. Check out Congrelate, another project of ours. Congrelate lets you view, sort, filter and share information about members of Congress. We've taken data from the Center for Responsive Politics, the Census, GovTrack and our own API and made it so you can create queries on that data and incorporate it into a big sheet. Of course, you can also export the data into CSV and JSON if you'd like, too.
Continue reading
59063
