Earlier this week the annual Law Via the Internet conference was hosted by the Legal Information Institute at Cornell University. The conference schedule featured talks on a range of policy and technical subjects, including the topic of extracting legal citations from text and understanding them programmatically, which arises whenever people need to determine the relevance of legal documents based on the authorities they cite. Recognizing citations in text is also a vexing but fun programming challenge, so I was excited to see this issue figure prominently in at least four separate talks.
Continue readingOpen States API at Hacks for Democracy in Philadelphia
This month, the first Hacks for Democracy event has been ongoing in Philadelphia. A new hackathon idea, #hacks4d hopes to... View Article
Continue readingCheck Out the Open States Beta Site
If you don't read the Sunlight Labs blog as religiously as I do (you should!), you might miss that our Open States project now has a public beta site up and running.
Continue readingA New Face For Open States
Wonder what the Open States team has been working on since we finished our initial goal of providing information for all 50 states back in March? As promised, we've been focusing on a new OpenStates.org and expanding our API to support full text search and we're finally ready to show you the results.
If you head over to OpenStates.org now you'll see that we've released a beta version of our site, currently available for 20 states. The remaining states are on their way later this year, but we wanted to make sure we took our time and did things right. As you explore the site you'll see all of the information we've been making available via our API. You'll also notice some enhancements made in the last few months like full-text search and enhanced support for legislator photos and contact addresses.
Continue readingOpen Data and the 2012 Elections: A Look at the VIP Hackathon
The day after TransparencyCamp, Sunlight joined the good folks at the Voting Information Project to host a hackathon. It was... View Article
Continue readingOpen States: 50!
Three years ago at PyCon 2009, we had the first PyCon Open Government Hackathon. Our big project was Open States (then the 50 State Project). The goal was to begin scraping state legislatures' websites in the hope of providing a common format for bill metadata across all 50 states.
Today, as we kick off the 4th Annual Open Government Hackathon at PyCon we're extraordinarily happy to announce one of the most significant milestones in the history of Open States: as of today, all 50 states (as well as DC and Puerto Rico) are now supported via our API and bulk downloads. This makes Open States the first and only completely open, completely free resource for accessing legislative information in a uniform format across all 50 states.
This is a proud day for all of us here and for everyone who has contributed to the project. Over the past three years Open States has grown to be much more than we'd envisioned and a great deal of that is due to great suggestions, contributions, and uses by the entire Open States Community. It is no coincidence that Open States has become Sunlight's most contributed-to open source project; we needed the community to make this project happen, and over 40 of you have answered the call.
Continue readingFollow State Legislatures with the New Open States iOS App
Today the Sunlight Foundation launches our Open States iPhone and iPad app that puts the inner-workings of state legislatures in the palm of your hand.
Continue readingHelp Open States Rate State Websites
As Open States closes in on our initial goal of supporting all 50 state legislatures (just 3 more to go!) we're also planning to put out a report card evaluating state legislative data across every state.
As the 40+ individuals that have sat down and helped us scrape state sites can affirm, most states simply don't do a decent job of making legislative information available so we're hoping that this can serve as a sort of wake up call to states that make this vital data far too difficult to access. For those few states that are doing a good job we're hoping to praise their commitment to open data and point out areas where they can do better.
We've come up with a set of criteria based on Sunight's "Ten Principles for Opening Government Data" (which expand upon the 8 Principles of Open Government Data) that we feel we can fairly apply to the states and created a survey to evaluate states against this criteria.
In order to guarantee a high quality report we'd like to get several responses per state and that's where you can help us out. Click the link below to head to a form that will ask you to evaluate the information that your state legislature makes available via their official website. By doing this you'll help us ensure that our eventual report is as accurate and as complete as possible.
(If you have any questions feel free to contact jturk@sunlightfoundation.com. If there are any questions you aren't sure how to answer we'd prefer you leave them unanswered instead of guessing.)
Continue readingIntroducing python-sunlight
Hello, World!
We'd like to welcome python-sunlight into the most excellent family of open-source projects maintained by Labs. This particular project aims to unify and normalize the Sunlight APIs into a single Python library that's easy to understand, use, and fun to play with.
This library currently supports our Congress API, Open States API, and Capitol Words API. As such we're deprecating the old python-sunlightapi and python-openstates libraries. They'll still work but will no longer be receiving updates, so switching is highly recommended.
This library has some neat features that should make migration painless - as well as some new features, such as a standardized location to place your Sunlight API Key, which makes testing (as well as distributing) your app even easier.
We've just released version 1.0.1 over on PyPI, which makes installation a snap on any system with pip
. The documentation is fairly complete, but feedback is super welcome -- we're eager to learn where folks get stuck.
Most of the bugs seemed to be worked out after the Boston Python Project Night, where we had a few folks test out the library. A special thanks to all our beta-testers!
Alright, so how do I get started?
Hacking on python-sunlight is super easy. Here's how to get setup.
You'll need an API key. If you've not done so, get an API key (it's alright, we'll wait, go ahead).
Back already? Great.
Now, you'll have gotten the email that has a long-ish string of letters and numbers - let's save this to ~/.sunlight.key
(where python-sunlight will look for a key). If you already had a key, it'd be worth it to go and dig it up.
If you're on a UNIX-type (MacOS, GNU/Linux, *BSD, AIX or Solaris (or any of the other POSIX-ey systems)) machine, you should be able to run a command that looks like the following:
echo "your-api-key-here" > ~/.sunlight.key
It's worth mentioning that your-api-key-here
should actually be your API key that was emailed to you up above.
Next, you should install python-sunlight via pip
. If pip
is not installed on your system, please download and install pip.
pip install sunlight
And you're good to go!
Without further ado, an example!
#!/usr/bin/env python
# Copyright (c) 2012, BSD-3 clause, Sunlight Labs
from sunlight import capitolwords
from sunlight import congress
phrase = "death metal"
# Today, we'll be printing out the Twitter IDs of all legislators that use
# this phrase most in the congressional record.
for cw_record in capitolwords.phrases_by_entity(
"legislator", # We're getting all legislators
sort="count", # sorted by how much they say
phrase=phrase, # this word
)[:6]: # We'll just try the top 5 legislators
legislator = congress.legislators(
bioguide_id=cw_record['legislator'], # Look up this biogude (unique ID)
# for every fed. legislator
all_legislators="true" # search retired legislators
)
if len(legislator) >= 1: # If we were able to find the legislator
legislator = legislator[0] # (this is a search, so it's a list)
if legislator['twitter_id'] != "": # and they have a Twitter ID
print "%s. %s (@%s) said %s %s times" % (
legislator['title'],
legislator['lastname'],
legislator['twitter_id'],
phrase,
int(cw_record['count'])
) # Print it to output :)
The output looks like this:
Sen. Feingold (@russfeingold) said death metal 979 times
Rep. Jackson Lee (@JacksonLeeTX18) said death metal 923 times
Sen. Leahy (@SenatorLeahy) said death metal 800 times
Sen. Kyl (@senjonkyl) said death metal 755 times
Sen. Durbin (@SenatorDurbin) said death metal 593 times
And once more (this time, searching for "san francisco"):
Rep. Filner (@CongBobFilner) said san francisco 1346 times
Sen. Feinstein (@senfeinstein) said san francisco 1288 times
Sen. Boxer (@senatorboxer) said san francisco 1181 times
Rep. Pelosi (@NancyPelosi) said san francisco 1135 times
Rep. Eshoo (@RepAnnaEshoo) said san francisco 677 times
Rock on!
Questions, concerns, bugs, patches, examples and virtual hugs are all welcome on our GitHub page, so please do check it out!
Continue readingOpen States Source Visualized
Open States recently reached a milestone in that we now support 40 states (and DC and Puerto Rico) and at our current pace we'll reach our goal of all 50 states by sometime early next year. It is only due to the fantastic support of our community and indviduals who have showed up at hackathons or just started contributing on their own that this goal is now in sight.
I thought it might be fun to look back on how the project has grown, and luckily gource is a piece of software for visualizing the history of a repository can help do just that. Watch below to enjoy a visually stimulating look back through the last two and half years of commits to Open States. You'll see flurries of activity around our hackathons, the drastic increase in activity from 2009 to 2010 and how 2011 so far takes up more than half the video, and some of the big refactors that we've made along the way to scale the project to a size well beyond what we initially conceived of.
Continue reading