Blog

Stay up to date on Sunlight’s work in D.C., throughout the country and around the world, as well as the latest open government, transparency and technology news.

Open States: Transparency Report Card

Today we’re making available our Transparency Report Card, a byproduct of the work we did in producing Open States.

 

Transparency Report Card

In the course of writing scrapers for all 50 state legislatures, our Open States team and volunteers spent a lot of time looking at state legislative websites and struggling with the often inadequate information made available.  Impossibly difficult to navigate sites, information going missing and gnarly PDFs of tabular data have become daily occurrences for those of us working on Open States. People are always curious to know how their state stacked up compared to others -- in fact one of the most frequent questions we have been asked has been “so which state was the worst?”  That question got us thinking:  How could we derive a measure of how “open” a state’s legislative data was?

After some consideration, we came up with six criteria on which each state could be evaluated, based on six of the Ten Principles for Opening Up Government Information: completeness, timeliness, ease of access, machine readability, use of commonly owned standards and permanence.  We omitted four of the original ten criteria (primacy, non-discrimination, licensing and usage costs) that tended not to present serious differences between states.

Evaluating each state on each criteria was a large task, and with community support we ensured that each state was evaluated by multiple people.  After the evaluation was complete, we converted the qualitative data on how a state performed to numeric scores (specific scoring details are available on the report card itself).  After summing these scores, states were also assigned a letter grade according to where they fell among their peers.  A state with a net score below negative one was given an F, a negative one or zero became a D.  With the average total score among states being a 1.5, we gave states with a net score of one or two a C, three became a B, and four and above became an A.

The final breakdown was 8 As, 11 Bs, 20 Cs, 6 Ds, and 6 Fs.   If you’re interested in how your state did compared to others you can check out all the details on the Open Legislative Data Report Card.

We know first-hand from our ongoing dialogue with state legislatures and open government technologists that identifying these commonplace problems can go a long way toward addressing them. In that spirit, and in the spirit of Sunshine Week, we offer this report card and recommendations today.

New Districts in Sunlight Congress API

Good news if you were one of the users waiting on our Congress API to support the newly drawn congressional districts! As of today it is possible to pass the districts=2012 flag to the Congress API's districts.getDistrictFromLatLong method to instruct the API to return the district in effect for the 2012 elections.

As you may recall, the data wasn't previously available in a uniform format but thanks to a recent data release from Census.gov we were able to get this data loaded, with days to spare until the election.

The default will remain to return the districts in effect for purposes of representation until the swearing in of the 113th Congress in January 2013 at which point the temporary districts=2012 flag will be retired (but it will be safe to continue to pass the parameter indefinitely).

This change does not yet impact other Sunlight API methods. The Open States district methods and the ZIP code related methods will be updated as that data is available, as described in our last update.)

Read more

A New Face For Open States

Wonder what the Open States team has been working on since we finished our initial goal of providing information for all 50 states back in March? As promised, we've been focusing on a new OpenStates.org and expanding our API to support full text search and we're finally ready to show you the results.

If you head over to OpenStates.org now you'll see that we've released a beta version of our site, currently available for 20 states. The remaining states are on their way later this year, but we wanted to make sure we took our time and did things right. As you explore the site you'll see all of the information we've been making available via our API. You'll also notice some enhancements made in the last few months like full-text search and enhanced support for legislator photos and contact addresses.

Read more

What Redistricting Means For Sunlight's APIs

Gerrymandering political cartoon

With the election 4 months away we're starting to get questions about when our various APIs and projects that depend upon them will return the newly redistricted legislative boundaries.

The short answer is that we will most likely not be able to support new district boundaries until after the November 2012 election (but before they technically go into effect for the purposes of representation in 2013).

If you're interested in the reasoning behind this decision, read on, but be warned, redistricting is a lot more complex than "the state has released new boundaries, let's load them."

Read more

Open States: 50!

Three years ago at PyCon 2009, we had the first PyCon Open Government Hackathon. Our big project was Open States (then the 50 State Project). The goal was to begin scraping state legislatures' websites in the hope of providing a common format for bill metadata across all 50 states.

Today, as we kick off the 4th Annual Open Government Hackathon at PyCon we're extraordinarily happy to announce one of the most significant milestones in the history of Open States: as of today, all 50 states (as well as DC and Puerto Rico) are now supported via our API and bulk downloads. This makes Open States the first and only completely open, completely free resource for accessing legislative information in a uniform format across all 50 states.

This is a proud day for all of us here and for everyone who has contributed to the project. Over the past three years Open States has grown to be much more than we'd envisioned and a great deal of that is due to great suggestions, contributions, and uses by the entire Open States Community. It is no coincidence that Open States has become Sunlight's most contributed-to open source project; we needed the community to make this project happen, and over 40 of you have answered the call.

Read more

Help Open States Rate State Websites

As Open States closes in on our initial goal of supporting all 50 state legislatures (just 3 more to go!) we're also planning to put out a report card evaluating state legislative data across every state.

As the 40+ individuals that have sat down and helped us scrape state sites can affirm, most states simply don't do a decent job of making legislative information available so we're hoping that this can serve as a sort of wake up call to states that make this vital data far too difficult to access. For those few states that are doing a good job we're hoping to praise their commitment to open data and point out areas where they can do better.

We've come up with a set of criteria based on Sunight's "Ten Principles for Opening Government Data" (which expand upon the 8 Principles of Open Government Data) that we feel we can fairly apply to the states and created a survey to evaluate states against this criteria.

In order to guarantee a high quality report we'd like to get several responses per state and that's where you can help us out. Click the link below to head to a form that will ask you to evaluate the information that your state legislature makes available via their official website. By doing this you'll help us ensure that our eventual report is as accurate and as complete as possible.

Fill Out a State Survey

(If you have any questions feel free to contact jturk@sunlightfoundation.com. If there are any questions you aren't sure how to answer we'd prefer you leave them unanswered instead of guessing.)

Read more

Open States Source Visualized

Open States recently reached a milestone in that we now support 40 states (and DC and Puerto Rico) and at our current pace we'll reach our goal of all 50 states by sometime early next year. It is only due to the fantastic support of our community and indviduals who have showed up at hackathons or just started contributing on their own that this goal is now in sight.

I thought it might be fun to look back on how the project has grown, and luckily gource is a piece of software for visualizing the history of a repository can help do just that. Watch below to enjoy a visually stimulating look back through the last two and half years of commits to Open States. You'll see flurries of activity around our hackathons, the drastic increase in activity from 2009 to 2010 and how 2011 so far takes up more than half the video, and some of the big refactors that we've made along the way to scale the project to a size well beyond what we initially conceived of.

Read more

Labs Olympics 2011: How Is Babby?

For this year's Labs Olympics I was on an all-star team comprised of Aaron, Alison, Tim, and myself, better known as the Labs Olympics Winners (note: we did not win, this was just our team name). Alison has a young baby at home and Aaron was out during our first brainstorming session for the birth of his niece so it wasn't a big surprise that we wound up with a plan to make a sophisticated baby monitor. (It might come as even less of a surprise that we named it How Is Babby in honor of an infamous web meme.)

At first all we knew is that we wanted to use some random gadget or assortment of Arduino sensors to give geek parents a way to monitor their geek children, but it wasn't until we realized we had a spare Microsoft Kinect sitting around the office that we realized exactly how far we could take it.

Kinect

The Kinect is an impressive device, sporting 4 microphones, RGB and IR cameras, an additional depth sensor, and a motor that allows vertical panning. Getting the Kinect running on Linux is a fairly well documented process. We leaned heavily on instructions from the OpenKinect community, which worked pretty much without issue. After doing the usual

cmake, make, make install
dance, things worked without issue on Ubuntu 11.04.

Also included in the OpenKinect source tarball are bindings for a half dozen languages, including Python. Having a Python wrapper made things incredibly easy to experiment with as I had access to python OpenCV bindings for displaying image data and NumPy for manipulating the matrices that the Kinect driver returns.

With these tools in hand we just had to decide what we actually wanted to get from the Kinect. We decided to take regular snapshots to present via a web interface, and also have a mechanism for the Kinect process to notify the web client when there was motion. Snapshots were extremely easy: with just a single line of code, we were able to bring back the RGB image from the Kinect's main camera and convert it to a suitable format using OpenCV. Once we made the discovery that there was also the option to bring in an IR image, we added a night-vision mode to our application as well. This way, the parent can adjust the camera to either take a standard image in normal light situations or switch to the IR camera for the night. (Due to a hardware limitation of the Kinect, it is impossible to use the RGB and IR camera at the same time.)

Given the uncertainty in the amount of available light and the fact that the depth sensor provided simpler data to work with (essentially a 2D matrix of depth values refreshed about 30 times per second), we decided to use the depth sensor to detect motion. NumPy's matrix operations made this a breeze. By averaging the depth of the frame and comparing the deviation across a range of frames, we could flag each individual frame as likely containing motion or not. Depending on the desired sensitivity of the alerts, the application would wait for anywhere from ten to thirty frames of consecutive motion before notifying the web application that the baby was on the move.

The Web Application

As opposed to a traditional baby monitor, which has a dedicated viewing apparatus, we liked the idea of a web console that could be viewed from anywhere, including via a mobile device. The main features of the web app would be viewing, motion alerts, and configuration of features such as SMS notifications and nightvision. The basic web app was built with Django, but we used a few add-on libraries to help accomplish our goals in the two days given for the contest.

We decided that the easiest way to get images to the user was to have the web page embed a single image that the monitoring software would update at a set interval. We used Socket.IO for a very light-weight solution to keep the image updated to the latest version. In the best case scenario, i.e. the user's browser supports it, Socket.IO will use WebSockets to keep the connection open, but will degrade gracefully and fall back to AJAX or other means to get the job done.

Because our team lacked a designer, we used a CSS framework to take care of cross-browser issues and provide some pre-designed UI elements. Twitter just recently released their Bootstrap framework, so we went with it. It styled all of the UI elements on our site, including a navigation bar, alert boxes, buttons, and a form. Although we had some unresolved trouble with the form elements not lining up properly with their labels, it proved very easy to work with, overall.

The remaining technical component of the website was the AJAX alerts on motion events detected (and logged in a DB table) by the backend. There were a few criteria for how it needed to work, the most important being that alerts needed to be somewhat persistent to the user, so that a user couldn't miss an all-important alert saying that the baby was moving, just because they were clicking quickly between pages on the site, for instance. This meant that we needed something more sophisticated than Django's inbuilt messaging framework (django.contrib.messages). The answer came in the form of django-persistent-messages. It was built to work right on top of Django's messaging system, so it worked seamlessly and was a no-brainer to set up. With django-persistent-messages working, alerts now would not disappear unless dismissed by the user, hopefully averting any potential baby-on-the-move mishaps.

In the end, there were a few features we had to leave unfinished to get the project out the door on time, including audio monitoring and SMS messaging, but we were pretty happy with the results. As usual, all of our code is available on GitHub: How Is Babby.

Read more

Open States API: 1 Year Later

Last September we announced the first public release of the Open States API. The API enables programmatic access to all of the key artifacts of the state legislative process. The API currently provides a standard interface to bills, votes, legislators, committees, and events across 36 states, Washington DC, and Puerto Rico.

Seeing as it has been a year since this first public release it seems like a good time to check on on where we are today and where we're going next.

Read more

Open States Reaches Halfway Mark

Open States map @ 25

Today marks an important milestone for the Open State Project: the addition of New York to our list of experimental states brings our total number of supported states to 25 (plus Washington DC). This marks the halfway point on our journey to bring clean, consistent, machine readable legislative information to all 50 states.

This means that residents of 25 states (accounting for approximately two-thirds of US citizens) can access their state's legislative data in a variety of machine readable formats (JSON, XML, CSV) and will soon be benefiting from sites like like OpenGovernment.org and MyGov365 that use our bulk downloads and free API to keep citizens informed about their state legislature.

Read more