In this third segment on unveiling opengovernment.org and how you can use it to track legislative issues, we are going... View Article
Continue readingOpen States API: 1 Year Later
Last September we announced the first public release of the Open States API. The API enables programmatic access to all of the key artifacts of the state legislative process. The API currently provides a standard interface to bills, votes, legislators, committees, and events across 36 states, Washington DC, and Puerto Rico.
Seeing as it has been a year since this first public release it seems like a good time to check on on where we are today and where we're going next.
Continue readingOpen States Reaches Halfway Mark
Today marks an important milestone for the Open State Project: the addition of New York to our list of experimental states brings our total number of supported states to 25 (plus Washington DC). This marks the halfway point on our journey to bring clean, consistent, machine readable legislative information to all 50 states.
This means that residents of 25 states (accounting for approximately two-thirds of US citizens) can access their state's legislative data in a variety of machine readable formats (JSON, XML, CSV) and will soon be benefiting from sites like like OpenGovernment.org and MyGov365 that use our bulk downloads and free API to keep citizens informed about their state legislature.
Continue readingSunlight Labs & Google Summer of Code 2011
We're proud to announce we've been accepted as a mentoring application for the Google Summer of Code 2011.
If you aren't familiar with Google Summer of Code, it is a great opportunity for college students and open source organizations to work together. Google pays students a $5000 stipend in exchange for their work on an eligible project. For more details about the program in general visit the GSoC 2011 website.
This is our third year participating and we're looking forward to another great summer and a new batch of students and projects.
Continue readingSunlight @ PyCon 2011
Two years ago we held an Open Government Sprint at PyCon 2009. We had never hosted an event like that before, and had no idea what to expect. To our amazement we ended up with one of the largest groups of any of the sprint projects, completely filling our room for the first few days. Approximately 30 people attended and kicked off what has now become the Open State Project.
Next week, we'll be heading to PyCon and hosting an Open Government Hackathon for the third year in a row. The primary focus will again be the Open State Project but our space is open to everyone interested in government data. If you have a project you'd like to hack on let us know and I'll be sure to mention your project when I plug the sprint. If you aren't attending PyCon but happen to be near Atlanta you're welcome to join too, the Hackathon is free and open to the public (March 14th-16th @ the Hyatt Regency in downtown Atlanta).
Additionally, I'm going to be presenting a poster on the technical aspects of the Open State Project on Sunday. I'll be around to talk about the project itself but also web scraping and opening government data in general, so if you're at PyCon stop by during the poster session Sunday morning and say hi.
Continue readingOpen States: Present and Future
I'm pleased to say that Caitlin and James have just finished giving our Open States project a lovely new design. Not only is the site now much more pleasing to look at, it's much easier to see the great progress that's being made by James, Mike and our volunteer contributors. In addition to the five states that are live (and supported by OpenGovernment), there are already another twelve states with "experimental" status. Don't let the scare-quotes scare you, though: while we wouldn't yet recommend building your air traffic control system or pacemaker firmware in such a way that it's dependent on our API coverage of Alaska, the scrapers from the experimental states are well on their way to being declared complete. Developers should confident about building around this data -- rest assured that it'll be declared "ready" soon enough.
Of course, we hope that developers in our community will also consider becoming involved in the project directly -- there's plenty of work to be done.
And it's genuinely important work. State legislatures are where vital decisions are made about civil rights, transportation, education, taxes, land use, gun regulation, and a host of other issues. Far too often, these issues don't get the attention they deserve. It's a simple question of scale: there are a lot more resources available at the federal level for both lawmakers and journalists. That means state governance both requires more transparency and tends to get less of it. We think technology can help make the situation better -- that's what Open States is all about.
There are some interesting opportunities for cross-state work, too. Polisci geeks will probably appreciate the comparative politics opportunities that a common data model and API will allow (Gabriel Florit's already been creating some cool visualization experiments that build on our data). But there are also less academic applications for this information. Consider these two stories that NPR published last fall. They got a bit lost in the pre-election shuffle, but they made a big impression on me.
The gist of it is this: Arizona's controversial immigration law didn't happen by magic. One of the special interests fighting for it was the private prison lobby -- as you might imagine, having more prisoners means more business for them, and they saw increased enforcement of immigration laws as a growth opportunity. So, via an intermediary organization that specializes in this sort of thing, they conducted a legislator "education" campaign, wining and dining lawmakers and sending them home with prewritten model legislation.
All of this is perfectly legal. And, depending on your opinion about immigration, you might even approve of the policy outcome it produced. But it's hard to imagine anyone being okay with the shadowy role that commercial interests appear to have played in this legislative process. If we'd been able to spot the provenance of the legislation earlier, would journalists and organizers have been able to give the people of Arizona a more complete understanding of what was going on? I think so -- I hope so. That's the kind of use that Open States should make possible, and the one I'm most excited about.
Continue readingNew Hampshire Opens Its Legislative Data
As recently covered on TechPresident, the New Hampshire General Court (their state legislature) has made an extremely welcome addition to their website in the form of a downloads section.
New Hampshire isn't the first state to offer such a thing: New Jersey has a similar section on their website, and quite a few states like New York and Kansas are introducing APIs to their new legislature websites. What is interesting, however, is the fact that the justification for offering the data presented by freshman representatives George Lambert and Seth Cohn is centered around reducing cost and strain on the legislature's website caused by web scrapers.
The load placed on sites by scraping them is something that we know a little bit about. Our Open State Project is currently crawling 18 state legislatures once a day, hitting over 100,000 pages nightly. Bulk downloads like New Hampshire's make it possible for us to take in all changes by simply downloading a few files every night instead of hitting thousands of pages--most of which haven't changed. Even though we take precautions like rate limiting our scrapers and having them back off if the site seems to be failing, we still see the occasional failure during our scraping run, which unfortunately only causes us to have to run the scraper again.
New Hampshire and its citizens will see other benefits of the bulk data beyond a less-burdened website. Consumers of the data will now be able to take the data in much faster than they previously could. There's also a much smaller potential for errors when you are importing data from a machine readable source like a CSV or database file. This means that tools built on top of scraped data (like the recently launched OpenGovernment beta) will be able to have more accurate and up to date data.
Those responsible for making this change happen in New Hampshire should be proud of the change that they've enacted. A preliminary glance at the actual New Hampshire data makes it look promising. As the data is quite new unfortunately they are not yet including roll call votes or links to the full text of bills, but we'll reach out to them to see if these oversights can be fixed in the near future. Hopefully New Hampshire is just one of many states that will start seeing the benefits of providing bulk access. To help show what is possible we'll be adding New Hampshire support to the Open State Project as soon as possible.
Continue readingOpenGovernment Is A Finalist For SXSW Accelerator
We've been really pleased with the enthusiastic response that OpenGov has received since its launch. Today's brought one more bit of good news: the site's been named a finalist in the Accelerator competition of this year's SXSWi. You can find the full details over at the OG blog. Congratulations to David, Carl and the rest of the PPF team!
And while I've got you thinking about state legislative data, check out what Gabriel Florit's done to visualize data from Open States. It's very neat stuff, and a great early example of the kinds of things that we think the project will make possible.
Continue readingWe Have a Winner: Contributor Raffle Update
Last October we announced that to celebrate our 100th project on GitHub we'd be giving away a prize to one lucky contributor.
Earlier this week we drew the names from all of those who were eligible and entered and Brandon Lewis was drawn as the winner.
Continue readingReminder: Submit a Patch, Win a Kindle
Back in October we announced that to celebrate our 100th GitHub project we'd be giving away a Kindle (or Gift Certificate) to say thanks to one lucky contributor to our Open Source projects.
If you find yourself with a bit of extra time during this holiday season and want to help make government more transparent please take some time to contribute to one of these projects (or any of the 100 others). In order to be eligible be sure to fill out the entry form.
Continue reading