Today we're launching 6° of Corporations, a new micro-site that provides some insight into the complicated area of corporate identity. It may sound trivial, but uniquely identifying a corporate entity is not easy. For federal contracting data (like in USASpending.gov), DUNS numbers are used to (supposedly) uniquely identify a contractor. However, there are problems in not only how DUNS numbers are issued and maintained, but also with the agency's use of DUNS numbers. To help illustrate this, we’ve created a visualization that shows the relationship between company names and company DUNS numbers in USASpending.gov.
Continue readingOpen States Source Visualized
Open States recently reached a milestone in that we now support 40 states (and DC and Puerto Rico) and at our current pace we'll reach our goal of all 50 states by sometime early next year. It is only due to the fantastic support of our community and indviduals who have showed up at hackathons or just started contributing on their own that this goal is now in sight.
I thought it might be fun to look back on how the project has grown, and luckily gource is a piece of software for visualizing the history of a repository can help do just that. Watch below to enjoy a visually stimulating look back through the last two and half years of commits to Open States. You'll see flurries of activity around our hackathons, the drastic increase in activity from 2009 to 2010 and how 2011 so far takes up more than half the video, and some of the big refactors that we've made along the way to scale the project to a size well beyond what we initially conceived of.
Continue readingA Very Vigo Halloween
For yesterday's Labs Open House, we had Vigo the Carpathian greet our guests the only way he knows how -- by following them intensely with his iron gaze.
Read on to see how it was done, and how to set up your own.
Continue readingDon’t Forget: Our Open House is Tomorrow!
A gentle reminder: our open house is tomorrow starting at 6, and we'd love to see as many of you here as can make it. Beer has been ordered, candy is being acquired, and plans are afoot for a Kinect-powered haunted painting. In short: it's going to be great. RSVP, why doncha?
Continue readingPyCodeConf 2011: Snakes in an Infinity Pool
If there is one thing that I learned from PyCodeConf, it's that all conferences should be in Miami in October. And they should all feature parties at rooftop infinity pools. Aside from the fun, PyCodeConf had a great selection of speakers that showed the breadth of the Python community, from wedding web sites to scientific computing. Read on for an overview of the some of the talks that pulled at my heartstrings.
The slides and audio from all talks can be downloaded from the PyCodeConf site.
What makes Python AWESOME?
This talk by Python core developer Raymond Hettinger was one of my favorites. When working with a language on a day-to-day basis, it is easy to take features for granted. Iterators, generators, and comprehensions are things that seem simple at first, but allow you to do very complex operations in very little code. The new-ish with statement provides an elegant interface for resource management and separation of common set up and tear down code.
Physics is renowned for the beauty and elegance of it's theories and equations. It's these same principles that made me love Python. While the language is slower in gaining new features, you can be guaranteed that the implementation will be incredibly clean and consistent with the principles of the language.
Embracing the GIL
I was fortunate to see David Beazley give a GIL thrashing talk at PyCon and this talk was just as good. The GIL is a very controversial part of Python which has both FUD and actual issues surrounding it. David has done a lot of research into how the GIL works and demonstrates how it behaves under various conditions. The summary: Python 2.7 is okay, Python 3 needs work, and a basic implementation of thread priorities in Python 3 puts it on par with 2.7.
API Design and Pragmatic Python
Kenneth Reitz is best known for his wonderful packages such as requests, envoy, tablib, and clint. If you've used any of Kenneth's projects you'll have noticed that he values creating sensible APIs that insulate users from the messier parts of Python. He takes a very conservative approach to his cause; no need to actually replace messy packages, just create wrappers that make them easier to use.
Kenneth also announced the release of The Hitchhiker’s Guide to Python. His goal is to create a central repository for Python best practices covering everything from installation and editors to coding style and app layout.
PyPy
The one common theme of nearly everything at the conference was PyPy, famed alterna-interpreter. The team has come a long way and everyone was eager to show the areas in which it excels over CPython and point out the parts that need some work.
The general consensus seems to be that over the next few years PyPy will become the interpreter of choice for running Python. The team is currently accepting donations on their site for general development, Python 3 support, and a port of NumPY. I've donated, you should too!
Who's coming with me next year?
I highly recommend checking out out each of the talks. Even though I only highlighted a few here, they were all quite excellent. Thanks to GitHub for putting on such a great conference and all of the sponsors that allowed it to happen (free mojitos).
Continue readingIn Depth with Campaign Finance Data
Influence Explorer and TransparencyData are the Sunlight Foundation’s two main sources for data on money and influence in politics. Both sites are warehouses for a variety of datasets, including campaign finance, lobbying, earmarks, federal spending and various other corporate accountability datasets. The underlying data is the same for both sites, but the presentation is very different. Influence Explorer takes the most important or prominent entities in the data--such as state and federal politicians, well-known individuals, and large companies and organizations--and gives each its own page with easy to understand charts and graphs. TransparencyData, on the other hand, gives searchable access to the raw records that make up each Influence Explorer page. Influence Explorer can answer questions like, “who was the top donor to Obama’s presidential campaign?” TransparencyData lets you dig down into the details of every single donation to that campaign.
Every chart and figure in Influence Explorer is derived from the detailed records in TransparencyData. But correctly computing totals from the raw records isn’t always straightforward. This article will explain how the figures on Influence Explorer are computed from TransparencyData records, and along the way show some of the intricacies of campaign finance rules and limitations of the data.
Continue readingLabs Olympics: Rex the Cleanosaur
We've got a problem here at Sunlight. A tiny kitchen and a large staff is a recipe for a disaster of immense magnitude. Each day a member of staff is assigned to kitchen duty; the list on the fridge has a schedule of who is responsible for keeping the dishwasher full and run as needed, wiping down the counters and making sure that dishes are put in the appropriate cabinets. Some staff members do a really great job, others are inconsiderate assholes who let the kitchen go to pot (I admit to being one of the inconsiderate assholes on occasion). We're smart people, there has to be a better way to do this!
Rex the Cleanosaur is a better way.
For the 2011 Sunlight Labs Olympics, Team Awesome created Rex the Cleanosaur, a kitchen duty management application. Rex generates a schedule from current staff members and emails them with their assignments. If you receive an assignment for a day that you will be unavailable, just click the link provided in the assignment email to defer your duty. Another staff member will be automatically scheduled to take your day, but the next day they have kitchen duty, you'll do theirs. Just to keep everything fair, the web app makes deferments publicly available. Rex will chase down and eat habitual deferrers!
We also developed a tablet based interface that will be hung in the kitchen to display the person that has the current kitchen duty. But what if that person is doing a terrible job? How awkward it would be to have to talk to them face-to-face to ask them to do a better job. No worries! The table interface has a "nudge" button that, when tapped, sends the person responsible for the kitchen an anonymous, passive aggressive email telling them to get their act together. If they are doing a great job, you can "throw 'em a bone" to thank them for their excellent work.
The nudges and bones are used to calculate rankings of the best and worst kitchen duty. If the person responsible for kitchen duty has a high nudge count, you know that they will probably need some extra reminders throughout the day. Managers could even use nudge/bone counts as a factor in determining yearly raises! Okay, not really, but you can still berate people with high nudge counts for their lack of consideration for their fellow employees.
So what's next? We plan to add a way for the person with kitchen duty to rate the office as a whole on how well they held up their end of kitchen cleanliness (placing dirty dished in the sink rather than the dishwasher, etc). There are also some early plans for coupons, exemptions from kitchen duty that managers can give away as rewards for good work or to pay off employees that stumble upon secret evil plans their managers are working on.
Team Awesome consists of Chris Rogers, Drew Vogel, and Jeremy Carbaugh.
Continue readingSave the Date: Labs Open House October 25
Jeremy mentioned it in this week's labs update, but it's worth broadcasting it more loudly: we're having another Sunlight Labs open house! It's been about a year since the last time we did this. We had a great time with you all back then, and are looking forward to doing it again.
So! Please mark your calendars: we'll be opening our doors on Tuesday, October 25 at 6pm. Expect drinks, games, technology chit-chat and more than a little Halloween-themed nonsense.
If you think you can make it, do us a favor and RSVP here. We're looking forward to seeing you there!
Continue readingCorporate Accountability Data in Influence Explorer
Sunlight reporter Ryan Sibley published a short post today on various efforts of Koch Industries to influence environmental policy. That piece is a perfect example of the value of bringing disparate datasets together under one site. Ryan pulls in data on Koch Industries’ lobbying activities, EPA enforcement actions against the company and a Koch executive that sat on an EPA advisory committee–all from Influence Explorer and TransparencyData.
The story is based on some of the new data sets that we’ve added to Influence Explorer: EPA enforcement actions, public comments on federal rulemakings and corporate employees on federal advisory committees. I’ll give a brief overview of each.
Continue readingLabs Update: October 2011
We've had the first hint of fall here in DC with Metro switching from AC to heat. Grab some apple cider and a baked good while you read this brisk update from Sunlight Labs.
Eric Mill, International Man of Transparency
For the last two weeks, Eric has been in Santiago, Chile, working in the offices of the Fundacion Ciudadano Inteligente. There, he's been chipping in on an API over Chilean parliamentary data, imparting what knowledge of JavaScript and Android he can and learning about the open government scene in Chile.
Influence Explorer
Yet again, new data sets have been added to Influence Explorer and Transparency Data. This past week was the official launch of EPA violations, public comments on federal regulations and corporate employees that sit on federal advisory committees. The BP profile page is a great example of each of these new data sets. Andrew has also spent some time working on infrastructure improvements to allow us to keep our copy of the Regulations.gov data up to date.
Real Time Congress
Before his international travel, Eric worked on adding Senate Floor information to both the Real Time Congress API and to the Congress app for Android and continued development on an upcoming website around alerts of government data. Being the spontaneous person that he is, he made a spur-of-the-moment trip to the Senate Office of Public Records to fetch documents relating to Senate Political Fund Designees, which he'll digitize and release as a CSV after returning from Chile.
The Open States Project
The Open States team has continued to expand their coverage with the addition of Nebraska and Iowa. The team is also working to add more special sessions and several features utilizing the actual text of legislation. James will soon be heading up north to Boston so be on the lookout in the coming months for exciting Sunlight North news.
Sunlight Live / Datajam
Luigi has continued to plug away at Datajam, a new platform to power Sunlight Live. In addition to the core platform, Dan has been working on a modular comment and discussion system for events. If you've been hankering to run your own real-time, data-driven events, you won't have to wait much longer. Datajam will be well-documented and open-sourced for everyone to use by the end of the year.
Subsidyscope
Clearspending 2011 has launched! In addition to the launch, Kaitlin and Drew have been working on a corporate identifier project. The site will feature a visualization of the impracticality of continuing to use DUNS numbers as the de facto standard for identifying corporations. It is based on a 2D JavaScript port of the Traer processing library and the "nice branching" processing sketch by Natasha Harper and Katie Adee. Kaitlin is currently working on two presentations for the Association for Budgeting and Financial Management conference (super wonky) and one for Open Government Data Camp in Warsaw (super European).
K2
K2, the codename for our second Knight Foundation mobile app, is moving along nicely. The final name and logo are days away from approval, data has been finalized and scraped, and design is moving from comps to build out. Ryan, Caitlin, and I are quite excited about this app and hope to have something good to show you for next month's Labs Update.
SunlightFoundation.com
The big news on the Foundation front is that we are embarking on a rebranding, redesign, and content reorganization of the main SunlightFoundation.com web site. Ali is getting to work on comps and mood boards and I'll be looking at a reorganization of our blog content. In addition to the redesign, we've been pushing out new campaign pages. See our Super Committee page to find out what we've been working on to bring transparency to the committee.
Team Design
Now back to full staff, Team Design has been swamped with new work. Chris has been knocking out presentations for Sunlight speaking engagements and worked on design for a few new projects. Caitlin has been working on Capitol Words (I promise, it's coming soon!) and K2. Ali has been working on the design for Kaitlin and Drew's corporate identifier site, a new visualization for federal budget projections over time, updates to Influence Explorer, the redesign for Sunlight Foundation and a microsite for the Reporting Group for research they're doing on financial databases.
Team Management
Tom has been doing the usual odds and ends, wrapping up some grant proposals and writing some reports for grants we've been working on. It's all very glamorous! Somewhat less boringly, he's also getting ready to head to the Code for America Summit next week. The big news, however, was the launch of Superfastmatch, a joint project with Media Standards Trust. Superfastmatch is an open source tool for quickly finding overlapping text between documents in a huge corpora. We've got some great stuff planned for Superfastmatch so stay tuned.
Team Sysadmin
Between searching for an entrepreneur to marry and making scallion oil pancakes, Tim has been working to improve our core infrastructure. In the office Tim has been buying new hardware to replace the switches we've had since we were a very small organization. Once the new hardware is installed and the chains of wireless routers are removed, we hope to have dead-zone free wireless access throughout the office. We now have data-filled Munin graphs for all of our servers. Tim also went to a Varnish meet-up at the SURGE conference.
Tidbits
- Aaron has been helping our reporting group make improvements to their site that will allow users to keep up with electronic FEC filings, particularly those from presidential candidates and members of the Super Committee.
- Our mobile game, Decipher DC, is on hold for the moment as Daniel works on an analytics and metrics dashboard.
- I'm headed to sunny Miami, FL tomorrow for PyCodeConf. Send me a message if you are going so we can meet up!
- The Labs is holding an open house later this month, so keep an eye on the blog here for information. We'll post invites once we figure out the maximum number of lasers and dry ice we can buy.
- Goodbye, 1800 Cafe! You've fed us well with delicious scrapple, egg and cheese sandwiches since we've been here in Dupont Circle.
59063
