As stated in the note from the Sunlight Foundation′s Board Chair, as of September 2020 the Sunlight Foundation is no longer active. This site is maintained as a static archive only.

Sunlight Foundation

Follow Us

From Sea to Shining Sea: us for Python

by

I am an extremely lazy person. I started on a new project recently that required me to delve into state and census tract data. The thought of the effort involved in locating and copy-and-pasting a dict mapping US state abbreviations to FIPS codes was so overwhelming that I just wanted to go take a nap instead. And once I got the FIPS code dict, I'd have to use it to generate URLs for state shapefile downloads. Ugh!

So instead of (yet again) copying a dict from some other source, I decided to do something more permanent. us, the result of my laziness, is a Python package that contains all sorts of state meta data in an easy to use API.

Continue reading

The Tech Behind Call on Congress

by

Call on Congress is the Sunlight Foundation's new free service to help anyone with a telephone connect with and learn about their lawmakers. Simply dial 1-888-907-6886 to find out how your representatives are voting on bills and raising campaign money and learn about upcoming bills in Congress.

While Sunlight prides itself on creating powerful, online tools that make government more transparent and accountable, we do so knowing that there is a segment of the public without access to the internet that we will never reach. Call on Congress is our attempt to bring information about Congress to anyone that can use a phone.

Twilio

The service is built on top of Twilio, a simple cloud communications API that enables us to create interactive phone and SMS applications. Users can navigate through a series of menus, be connected directly to the office of their representatives, and even leave us voice messages.

I can't express how awesome it is to work with their service. They have made it incredibly easy to do things that would otherwise be quite difficult to create. We can generate dynamic responses that are spoken back to the user using Twilio's text-to-speech feature. We can have recordings of a spoken script play for static content or even play music, if we had the need. We can even forward phone calls so that users can be connected directly to the office of their representative without having to write down the phone number and dial the call.

Twilio also has SMS features that, while not used on this project, are used on our other new service, Scout.

i18n and the Google Translate API

Call on Congress is Sunlight's first multilingual project. The service is available in English, Spanish, and Esperanto... yes, Esperanto. This posed some unique challenges as we had to deal with both a static spoken script and dynamic text that would be read using Twilio's text-to-speech feature.

Dan created an amazing, multi-tiered solution for this problem. Each bit of text used in the application is run through a series of steps to determine the proper response. All of the text is written in English in the code, but is internationalized by:

  • creating an MD5 hash of the text
  • determining the response language
  • checking for a cached translated copy of the text if it is not in English
  • generating a translated copy of the text using the Google Translate API
  • generating an audio file name based on the MD5 hash and a slugged chunk of the text
  • doing a HEAD request to S3 to see if an audio file exists for the chunk of text
  • responding either with a Twilio Say command using the translated chunk of text or a Play command with the S3 URL of the audio file, if one exists

This method allows us to basically drop in new languages as we create the audio for them. We can launch a new language and have the text-to-speech render the Google translated text until we drop language audio files into the application.

Dan is smart.

Sunlight APIs

One of the most exciting aspects about this project, at least for us internally, is that almost all of the features of this service are powered by our existing APIs:

Our friends at TurboVote are responsible for the API that allows us to provide information about local election offices.

Call on Congress is a great example of what can be done using our API offerings. We were able to create this application without having to do any scraping or collecting of data of our own. In fact, it's a great example of what can be done entirely with APIs. Call on Congress is nothing more than a cache and the logic that combines the Twilio, Google Translate, and Sunlight APIs into a single application.

Give it a try! 1-888-907-6886

Call on Congress is open source and available on GitHub.

Continue reading

The Tech Behind TransparencyCamp

by

TransparencyCamp, Sunlight's open government unconference, is one of the few chances the Labs gets each year to go crazy with tech. Our goal is to use technology to enhance the conference experience and set the expectation for the type of "maker" culture we have here at Sunlight. Read on to find out some of the technology that makes TransparencyCamp run.

Web and Mobile Sites

Transparency Camp is somewhat of a hybrid unconference. A small number of sessions are planned in advance and we try to keep the session board as-is once it is initially set. One reason for this is that we have the sessions listed on the web site, mobile web app and screens at the venue. We just don't have the resources to constantly monitor the board for changes and have that reflected on all of the other places sessions are listed.

When someone submits a session, it is manually entered into the TCamp database and a physical print-out is placed on the schedule board. The TransparencyCamp codebase includes an undocumented API that provides feeds of all upcoming sessions as well as the full conference schedule. The backend service also pulls in tweets from Twitter that match event-related hash tags and messages from the official TCampDC account.

The mobile app is an HTML-based site that has been tested on both iOS and Android devices. The app was built on a long outdated version of Backbone.js that gets sessions, tweets and photos from the TransparencyCamp API. The social feeds are updated every minute so attendees can watch the stream as it happens.

Etherpad

Each year at TCamp we want to provide a way for attendees to take notes during sessions. Last year we turned session pages into mini-wikis where users could click to edit the page to add notes. The usage, as we mostly expected, was disappointing. The user would have to click edit, make their changes, hope someone else hadn't saved other changes in the meantime and then hit save. While not the most laborious process ever created, it was enough of a barrier to keep people from participating in note taking.

We've had great success with an internal instance of Etherpad here at Sunlight so Eric suggested we incorporate it into the TransparencyCamp site. If you are not familiar with it, Etherpad is a collaborative document editor much like Google Docs. We used the embeddable view and slapped a collaboratively editable document right onto each session page. Attendees could then immediately take notes without clicking around and without worrying about clobbering other people's changes.

We found that many more people participated in note taking and those that did had nothing but great things to say about the experience. Etherpad really hit the sweet spot of collaboration that a wiki just couldn't reach.

Optimizing Registration

It's the little things that count. Most people, when setting up a 4-lane registration table, would just divide last names by first letter into even groups of four. But what if there isn't an even distribution of last names? Andrew saw this potential inefficiency and sprung into action.

Armed with our list of registrants, he calculated the frequency of the first letter of the last names of attendees. The frequency results were then fed into a script that iterated through the possible partitions of the alphabet, selecting the partition that minimized the standard deviation of percentages of the alphabet of each partition.

View the code on GitHub.

Nicko approves of the optimized registration lanes. Photo by stereogab.

Photo Booth

While not necessarily new, Tim set up another instance of our Sunlight Photo Booth. It's really just an iMac running our web-based Photo Booth software, but use your imagination here. The HTML user interface communicates with the backend over a WebSocket connection, which invokes isightcapture to take each photo. A Python script then uses PIL to add a Lomo-esque effect to each photo and combine them into a single strip. The generated strip is then returned to the web-based UI, uploaded to Flickr and a QR code is displayed that links to photo strip's Flickr page. Whew!

2013?

We've already been discussing ideas for new tech at TransparencyCamp 2013: our own registration and payment processing system, wall-crawling robots to scan the schedule, RFID implants (badges, not people… okay, maybe people) and more. What will make the cut? Stay tuned!

Continue reading

Labs Update: May 2012

by

Like a phoenix rising from its ashes on a monthly basis, it's Labs Update time!

TransparencyCamp 2012

It may be cliché to say, but TransparencyCamp 2012 was the best TCamp ever! GROUP HUG! We doubled attendance from last year with over 400 attendees from 26 US states and 27 countries. Anything I write here won't do the awesomeness of the event any justice so just watch the video:

TransparencyCamp wouldn't have been possible without the effort and expertise of the entire Sunlight Foundation staff, but I want to highlight the work of our newest designer, Amy Cesal. Event branding was her first task here in the Labs and I think it's pretty clear that she knocked it out of the park. Great work, Amy!

Open State Project

Sunlight Boston got the chance to spend a week with us at the DC headquarters during TransparencyCamp. It was great having them in office, even if Paul is a tab zealot.

Paul and James have done a lot of work on the API side of the project, implementing full-text search and enhanced event support as it relates to committees and bills. Thom has been focused on getting the public site closer to launch, working on the new design with Ali and refining news/blog aggregation.

James also released a new version of scrapelib. The update features FTP and retry support, optionally obeying robots.txt and a pluggable caching layer. scrapelib is now based on requests, Kenneth Reitz's ubiquitous HTTP library.

Influence Explorer

It's non-stop data with the Influence Explorer team. Ethan worked to add Super PAC and independent expenditure sections on profiles. Alison processed updates to Contractor Misconduct data from POGO. Andrew did more work on the new regulatory filings section, which is planned to launch sometime in July.

Scout

Eric recently launched an open beta of Scout, an alert system for the things you care about in state and national government. It covers Congress, regulations across the whole executive branch and legislation in all 50 states.

You can read more details about the project in Eric's launch blog post, but here is a quick rundown:

  • notifications via email, SMS, RSS and JSON
  • searching for keywords and phrases in bills, speeches and regulations
  • detailed activity on specific bills

Scout is yet another new Sunlight project that is built almost exclusively on our public API services including Open States, Capitol Words and Real Time Congress.

Team Journalism

Ryan investigated the exciting and fast-paced world of tariff suspensions for a piece she wrote on the miscellaneous tariff bill process.

Lee has been running a grade level analysis of congressional speeches, which have been declining over the last seven years. The piece, which I hope scores higher than congressional speeches, should be published within the next week or two.

Jacob crunched third quarter independent expenditure numbers after monthly and quarterly filers posted results this month, is beginning work on a Party Time redesign to take place this summer and threw together a real-time FEC filing system monitor.

Open Source

Now that we are up to 186 open source projects on GitHub, I figure it's about time we feature the best of what we've got. Newly released projects include:

  • citation is a JavaScript library for extracting US Code citations from blocks of text. Eric has also provided citation-api, a small node.js wrapper to provide citation as a service.
  • bill-nicknames is a project to crowdsource popular names for bills. The goal is to map popular-but-unofficial names like 'Obamacare' to the official bills to which they refer.
  • oyster is a service for tracking regularly-accessed pages. It will cache pages that are frequently scraped, downloading new versions when page content changes.

Tidbits

  • Our pals at Cubox are working to get DataJam ready for public use
  • Daniel has been working on tools for the manual collection of political ad buy files at TV stations around the country.
  • Drew and Kaitlin have been working on SuperFastMatch and related tools, including a browser extension.
  • Ali and team have been designing for a bunch of projects including the new Open States public site, Sunlight Academy, the Sunlight Foundation redesign, Scout and Party Time.
  • Dan crunched numbers for a bunch of stories based on Capitol Words and has been looking into new technologies and data sets to be included in the project.
  • Tom has been helping to manage the third Knight app's progress, working on some new project proposals and desperately clawing his way out of a huge pile of email that accumulated during tcamp.
  • A Sunlight Olympics hack, but not the one mentioned in the post, has grown into a full project! We'll have more details next month and an announcement at Personal Democracy Forum in June.
  • May's album of the month is Threads by Now Now. I'm sure some of my coworkers may disagree, but they have no say in this post… so there!

Continue reading

Labs Update: April 2012

by

It's time again for our monthly Labs update. APRIL FOOLS! It's only been bimonthly so far this year! Oh, man, I got you so good.

Anyway, here's what we've been up to:

PyCon US 2012

Sunlight Foundation had a strong presence at PyCon consisting of Kaitlin, James, Thom, Paul and me. Our annual Open Government sprint didn't disappoint as we got a lot of participation on Open States and a new semi-announced project. This was also the first chance that Kaitlin and I got to meet Paul and Thom, confirming that James is in fact capable of hiring good people.

We also trekked through 20 minutes of office parks just to get to In-N-Out Burger. That's dedication.

Upwardly Mobile

screenshot of Upwardly Mobile

We finally launched! Check us out on your desktop, phone or tablet at http://upwardly.us. We've also created a launcher app for Android devices so that you can have a convenient icon on your home screen.

Upwardly Mobile is Sunlight's first responsively designed site. We learned a lot in the process and hope to use this method more often in the future.

Upwardly Mobile lets you find out where in the country it's best to live by comparing various types of salary, living and employment data and ranking it based on your preferences. While economics are not the only factor in the decision of where to live, the app can be useful to help you find locations that you may have not considered before.

K3, IDEO

Work has begun on the third and final app, codenamed K3, for our grant with the Knight Foundation. We are taking a more thoughtful approach with this app, working with the fine folks at IDEO on the purpose and design of the project. Tom, Ali, Anu and I spent a few days out in San Francisco with the IDEO team getting a feel for their process and working through goals for the app. I also took the opportunity to eat massive amounts of raw oysters and fennel; my belly misses you, San Francisco.

Sunlight and IDEO teams together

We are entering a research phase now to identify the data sets that will support the app ideas that have been proposed. From there it will go back to IDEO for wire-framing and interaction design, then back to us for execution. Launch is scheduled for fall!

Check out our other apps from the Knight Grant:

Influence Explorer

Influence Explorer now contains frequently updated campaign finance data from the FEC! High five to Ethan for all of his work on this. The new data shows up-to-date fundraising for candidates and PACs, as well as new tables for Independent Expenditures. Alison worked on new data updates for campaign contributions, federal grants and contracts, and federal advisory committee data so that IE stays as accurate and relevant as possible.

Andrew has mostly been toiling away on Sunlight's upcoming regulatory-comments-tracking site, but also snuck off to speak at Transparency Works last week in Vilnius, Lithuania.

OpenStates

It's been a long time coming, but OpenStates has reached all 50 states, DC, and Puerto Rico in either production or experimental status! This is a huge accomplishment and a testament to James' dedication, talent and lack of anything else to do. Document-related features will be coming to the API, laying the groundwork for much-awaited full text search and document comparison.

Paul and Thom have been working hard on data quality improvements to move the remaining states from experimental to production status. The team will soon begin work on a new web interface for OpenStates, opening up the API data for anyone to browse.

Semi-public Scraper Project

Daniel put together a mini-site for the not-yet-quite-ready-for-the-public scraper project that uses ScraperWiki to collect congressional information. We ran some initial trials of the project at PyCon and a ScraperWiki event here in DC. We'll be making a decision soon if this project will graduate from the experimental stage. If so, we'll be sure to announce it here.

Team Journalistic Integrity

Jacob has expanded the super PAC tracker to include all independent expenditures and electioneering communications, as well as all committees making independent expenditures or electioneering communications.

Ryan's had the opportunity to report on two national news stories -- the Trayvon Martin shooting and the Virginia ultrasound bill -- using the SuperFastMatch technology that is being developed by Media Standards Trust. She used the tool to identify states that had adopted model legislation written by special interest groups. Ryan has also been working on stories and projects related to campaign finance in order to reveal who really influences the way Washington works and how they do it.

Lee has been working with Planet Money and This American Life on the value of congressional committee assignments and figuring out how much lobbying lowers corporate tax rates. Check out This American Life episode 461 that features the work of Lee and the rest of the Sunlight Foundation!

Team Leadership

Tom has been doing some traveling: San Francisco the K3 kick-off meeting with IDEO, and Austin to join a panel at SXSW and eat some tacos (mostly the taco part). In between he worked with Ryan on her SuperFastMatch project. At the moment he's working on hiring and a project proposal; next week it's back to the west coast.

Team Retribution

Hellcats promotional image

Ali has spent much of the past two months undoing an organization-wide re-branding she secretly undertook late last year. Ali was a huge fan of the CW's cheerleading show Hellcats and spent many months creating themed templates for all of our sites. You should have seen the theme for the Labs site: Alice, standing at the side with her injured wrist, giving the evil eye to Lewis and Marti as they practice. She'll have to find another outlet to mourn the cancellation of the show though; it doesn't quite fit into our mission as a government transparency organization.

This is a lie. Ali didn't send me anything for this update so I got to make it up for her.

Tidbits

  • Tim just returned from China bearing gifts as consolation for any servers that broke during his absence.
  • Poligraft is about to get faster and more accurate thanks to swell work from Dan.
  • Capitol Words might be getting some new data sets soon. More on that in a future update.
  • Eric has been working on getting a new government information search and alert web site ready for launch.
  • Using technology developed from his compulsive need to track every item he purchases, Daniel has been working on a prototype of a bar code scanning app that would connect products to the political contributions of the companies that produce them.
  • The final six sectors of SubsidyScope have launched!
  • Our new designer, Amy Cesal, will be starting next week!
  • We are still hiring.
  • Album of the month: Threads by Now Now.
  • Smoothie of the month: black cherry, strawberries, cocoa powder, and avocado from Yola.

Continue reading

Upwardly Mobile and Responsive Design

by

Upwardly Mobile lets you find out where in the country it's best to live by comparing various types of salary, living and employment data and ranking it based on your preferences.

After months of "coming soon" teasers, we are finally launching Upwardly Mobile! The site uses economic data provided by the government to help you find areas of the country where you, based on your occupation, could be more economically secure than where you currently live. While economic concerns are only one part of the decision of where to live, Upwardly Mobile can help you find areas of the country that you may have never considered before or help you rule out those you have.

Upwardly Mobile resizes for phone, tablet, and desktop

Continue reading

Labs Update: February 2012

by

Previously in Sunlight Labs: Influence Explorer redesigned, James moved to Boston, and Capitol Words was released. So then why is Luigi cleaning out his desk? Where did Transparency Data go? Why is Ethan calling in to the morning check-in meeting? Find out on this episode of Labs Update!

Goodbyes…

Let's start off with some terrible news; Luigi Montanez has up and left the Labs. He'll be working for a new startup in the world of politics so I'm sure you will see much more of his amazing work in the near future. The local frozen yogurt and frozen custard shops will feel the loss of his business.

We also recently said goodbye to designer Chris Rogers. Did you like our Indecent Disclosure poster or the Transparency Camp 2011 branding? That's just two examples of her fine work.

...lead to open positions!

Sunlight Foundation is hiring!

If you haven't noticed, this is a really great place to work. Talented people, a fun environment, and lots of nearby delicious food. We also do really important work.

We're even offering a referral bonus: if we hire your suggestion, I'll give you a sincere hug.

Sunlight Seattle and Influence Explorer

Goodbye, Transparency Data! As part of an effort to streamline our branding, TransparencyData.com has been transformed into data.influenceexplorer.com. If you rely on the Transparency Data API, do not fret; it is the exact same API at a new URL and with much nicer documentation. All calls to Transparency Data URLs are being forwarded to the new domain so don't freak out thinking that all of your projects have broken.

In data-related news, Alison has recently refreshed state campaign finance data with the latest dump from FollowTheMoney.org. Ethan has wrapped up development on a project I hinted to at the end of last year where we will be pulling in timelier, but messier, data directly from the FEC. Andrew has been cleaning up much of IE's JavaScript infrastructure and working on a new site highlighting influence on the regulatory process.

Influence Explorer team lead Ethan has moved to Seattle, but will still be working remotely for Sunlight. Unfortunately, my knowledge of Seattle is quite limited so I've no witty jabs to include here. If you have any, please contact me.

Sunlight Boston and the Open States Project

It's like a spin-off of your favorite sitcom, but just as good as the original. Sunlight Boston is fully staffed with new hires Paul Tagliamonte and Thom Neale. Rounding out the team is a new data quality intern, Nina Psoncak. They're based out of a hip co-working space so if you find yourself in the area, stop by, compliment their code, and tell them how much cooler they are than Sunlight DC.

And they are getting work done too! Scrapers have been fixed and/or updated for California, Delaware, Colorado, Hawaii, and Rhode Island. billy, the underlying scraping system, now allows for the merging of legislators. The Open States API has been updated with several feature requests to better support mobile clients.

Sunlight Live

We took the State of the Union address as an opportunity to try out our new Sunlight Live platform, Datajam. Dan and Luigi (before his defection) did a really amazing job on the project. The event administration tools and chat module are super slick.

Upwardly Mobile

Upwardly Mobile cow

I've talked about it for months, but seriously, we are wrapping up Upwardly Mobile! The finishing touches are being added (animated cow, need I say more?), communications and organizing are planning the launch, and I'm wrestling with final tweaks to static maps generated with matplotlib.

Work begins on the third Knight app soon! We'll have an exciting announcement about a new partner that will be working on it with us.

python-sunlight

As the number of APIs we offer increases, so do the number of client libraries needed to work with each service. The madness must stop! Paul has started work on python-sunlight, a grand unified Python wrapper for (eventually) all of Sunlight's APIs. We are launching with support for the Capitol Words, Congress and Open States APIs. An experimental version of the Influence Explorer API is included and work on the Real Time Congress API will begin soon. Just pip install sunlight to get started. Python and Sunlight are BFF.

Subsidyscope

As the Subsidyscope project winds down, Kaitlin and Drew have updated the data with the latest release from USASpending.gov and prepared six more sectors for impending launch. They've also been working with Superfastmatch for some upcoming projects.

Team Journalism

The Sunlight Reporting group has historically been responsible for all journalistic output, but in recent months Labs has been taking an increasing role in our reporting. We've got access to these vast data sets, so why not do something worthwhile with them, right?

Joining the team is new hire Jacob Fenton who will be working as our embed in the Reporting Group. Since starting at Sunlight he's been knee deep in the swampy morass commonly known as raw FEC campaign finance reports. Ryan and Lee have been covering super PACs and elite donors for the 2012 presidential campaign.

Team Sysadmin

With our ever expanding troop of remote workers, Tim was tasked with finding a solution to replace our existing (and terrible) conference call system. Using ambient mics mounted in the ceiling, a mixer, a web cam, and Google Hangouts, Tim was able to rig up a solution that works surprisingly well!

our ceiling robot microphone

When the robot's nose glows green, you know you are being broadcast.

Team Tom

Tom has been (with much help from James and Daniel) setting the groundwork for a new scraper project (GASP!, literally) that we'll be asking you all to lend a hand with (stay tuned for more on that). Otherwise it's been the usual glamorous mix of contracts, grant reports, negotiating metrics and dealing with turnover. But he did buy a BeagleBone, which is kind of exciting (less so for Tim, who's been receiving a lot of tedious questions about rc.d and wpa_supplicant as a result).

Tidbits

  • Ever wondered if there are any crazy late night Tweets from members of Congress that get deleted the next morning? Eric is working on a project that will help uncover long lost updates from Congress.
  • Shouldn't come as a surprise, but there's been another awesome update to Congress for Android consisting of design updates and search features.
  • Daniel released Lapidus, a metrics tracking dashboard we will be using internally to track our goals.
  • February should see the release of the 180° project which will turn the cameras on the audience at Congressional committee hearings.

  • Ali has been slowly rolling out our new logo as the rebranding effort continues. The next few months should see updated identity on all of our properties including a completely new SunlightFoundation.com!

    new Sunlight Foundation logo

  • An awesome new release of the Congressional Roku app is coming soon.

  • February's afternoon snack of the month is the PBJ smoothie at Yola. Tell them Sunlight sent you and they'll look at you weird because they don't know who we are!
  • The sandwich gods are smiling upon Sunlight.

Continue reading

CFC (Combined Federal Campaign) Today 59063

Charity Navigator