TransparencyCamp, Sunlight's open government unconference, is one of the few chances the Labs gets each year to go crazy with tech. Our goal is to use technology to enhance the conference experience and set the expectation for the type of "maker" culture we have here at Sunlight. Read on to find out some of the technology that makes TransparencyCamp run.
Web and Mobile Sites
Transparency Camp is somewhat of a hybrid unconference. A small number of sessions are planned in advance and we try to keep the session board as-is once it is initially set. One reason for this is that we have the sessions listed on the web site, mobile web app and screens at the venue. We just don't have the resources to constantly monitor the board for changes and have that reflected on all of the other places sessions are listed.
When someone submits a session, it is manually entered into the TCamp database and a physical print-out is placed on the schedule board. The TransparencyCamp codebase includes an undocumented API that provides feeds of all upcoming sessions as well as the full conference schedule. The backend service also pulls in tweets from Twitter that match event-related hash tags and messages from the official TCampDC account.
The mobile app is an HTML-based site that has been tested on both iOS and Android devices. The app was built on a long outdated version of Backbone.js that gets sessions, tweets and photos from the TransparencyCamp API. The social feeds are updated every minute so attendees can watch the stream as it happens.
Etherpad
Each year at TCamp we want to provide a way for attendees to take notes during sessions. Last year we turned session pages into mini-wikis where users could click to edit the page to add notes. The usage, as we mostly expected, was disappointing. The user would have to click edit, make their changes, hope someone else hadn't saved other changes in the meantime and then hit save. While not the most laborious process ever created, it was enough of a barrier to keep people from participating in note taking.
We've had great success with an internal instance of Etherpad here at Sunlight so Eric suggested we incorporate it into the TransparencyCamp site. If you are not familiar with it, Etherpad is a collaborative document editor much like Google Docs. We used the embeddable view and slapped a collaboratively editable document right onto each session page. Attendees could then immediately take notes without clicking around and without worrying about clobbering other people's changes.
We found that many more people participated in note taking and those that did had nothing but great things to say about the experience. Etherpad really hit the sweet spot of collaboration that a wiki just couldn't reach.
Optimizing Registration
It's the little things that count. Most people, when setting up a 4-lane registration table, would just divide last names by first letter into even groups of four. But what if there isn't an even distribution of last names? Andrew saw this potential inefficiency and sprung into action.
Armed with our list of registrants, he calculated the frequency of the first letter of the last names of attendees. The frequency results were then fed into a script that iterated through the possible partitions of the alphabet, selecting the partition that minimized the standard deviation of percentages of the alphabet of each partition.
Nicko approves of the optimized registration lanes. Photo by stereogab.
Photo Booth
While not necessarily new, Tim set up another instance of our Sunlight Photo Booth. It's really just an iMac running our web-based Photo Booth software, but use your imagination here. The HTML user interface communicates with the backend over a WebSocket connection, which invokes isightcapture to take each photo. A Python script then uses PIL to add a Lomo-esque effect to each photo and combine them into a single strip. The generated strip is then returned to the web-based UI, uploaded to Flickr and a QR code is displayed that links to photo strip's Flickr page. Whew!
2013?
We've already been discussing ideas for new tech at TransparencyCamp 2013: our own registration and payment processing system, wall-crawling robots to scan the schedule, RFID implants (badges, not people… okay, maybe people) and more. What will make the cut? Stay tuned!
Continue readingLabs Update: May 2012
Like a phoenix rising from its ashes on a monthly basis, it's Labs Update time!
TransparencyCamp 2012
It may be cliché to say, but TransparencyCamp 2012 was the best TCamp ever! GROUP HUG! We doubled attendance from last year with over 400 attendees from 26 US states and 27 countries. Anything I write here won't do the awesomeness of the event any justice so just watch the video:
TransparencyCamp wouldn't have been possible without the effort and expertise of the entire Sunlight Foundation staff, but I want to highlight the work of our newest designer, Amy Cesal. Event branding was her first task here in the Labs and I think it's pretty clear that she knocked it out of the park. Great work, Amy!
Open State Project
Sunlight Boston got the chance to spend a week with us at the DC headquarters during TransparencyCamp. It was great having them in office, even if Paul is a tab zealot.
Paul and James have done a lot of work on the API side of the project, implementing full-text search and enhanced event support as it relates to committees and bills. Thom has been focused on getting the public site closer to launch, working on the new design with Ali and refining news/blog aggregation.
James also released a new version of scrapelib. The update features FTP and retry support, optionally obeying robots.txt and a pluggable caching layer. scrapelib is now based on requests, Kenneth Reitz's ubiquitous HTTP library.
Influence Explorer
It's non-stop data with the Influence Explorer team. Ethan worked to add Super PAC and independent expenditure sections on profiles. Alison processed updates to Contractor Misconduct data from POGO. Andrew did more work on the new regulatory filings section, which is planned to launch sometime in July.
Scout
Eric recently launched an open beta of Scout, an alert system for the things you care about in state and national government. It covers Congress, regulations across the whole executive branch and legislation in all 50 states.
You can read more details about the project in Eric's launch blog post, but here is a quick rundown:
- notifications via email, SMS, RSS and JSON
- searching for keywords and phrases in bills, speeches and regulations
- detailed activity on specific bills
Scout is yet another new Sunlight project that is built almost exclusively on our public API services including Open States, Capitol Words and Real Time Congress.
Team Journalism
Ryan investigated the exciting and fast-paced world of tariff suspensions for a piece she wrote on the miscellaneous tariff bill process.
Lee has been running a grade level analysis of congressional speeches, which have been declining over the last seven years. The piece, which I hope scores higher than congressional speeches, should be published within the next week or two.
Jacob crunched third quarter independent expenditure numbers after monthly and quarterly filers posted results this month, is beginning work on a Party Time redesign to take place this summer and threw together a real-time FEC filing system monitor.
Open Source
Now that we are up to 186 open source projects on GitHub, I figure it's about time we feature the best of what we've got. Newly released projects include:
- citation is a JavaScript library for extracting US Code citations from blocks of text. Eric has also provided citation-api, a small node.js wrapper to provide citation as a service.
- bill-nicknames is a project to crowdsource popular names for bills. The goal is to map popular-but-unofficial names like 'Obamacare' to the official bills to which they refer.
- oyster is a service for tracking regularly-accessed pages. It will cache pages that are frequently scraped, downloading new versions when page content changes.
Tidbits
- Our pals at Cubox are working to get DataJam ready for public use
- Daniel has been working on tools for the manual collection of political ad buy files at TV stations around the country.
- Drew and Kaitlin have been working on SuperFastMatch and related tools, including a browser extension.
- Ali and team have been designing for a bunch of projects including the new Open States public site, Sunlight Academy, the Sunlight Foundation redesign, Scout and Party Time.
- Dan crunched numbers for a bunch of stories based on Capitol Words and has been looking into new technologies and data sets to be included in the project.
- Tom has been helping to manage the third Knight app's progress, working on some new project proposals and desperately clawing his way out of a huge pile of email that accumulated during tcamp.
- A Sunlight Olympics hack, but not the one mentioned in the post, has grown into a full project! We'll have more details next month and an announcement at Personal Democracy Forum in June.
- May's album of the month is Threads by Now Now. I'm sure some of my coworkers may disagree, but they have no say in this post… so there!
Scout, in Open Beta
We're opening a new tool to the public today for beta testing, called Scout.
Scout is an alert system for the things you care about in state and national government. It covers Congress, regulations across the whole executive branch, and legislation in all 50 states.
You can set up notifications for new things that match keyword searches. Or, if you find a particular bill you want to keep up with, we can notify you whenever anything interesting happens to it -- or is about to.
Just to emphasize, this is a beta - it functions well and looks good, but we're really hoping to hear from the community on how we can make it stronger. You can give us feedback by using the Feedback link at the top of the site, or by writing directly to scout@sunlightfoundation.com.
Continue readingShouldn’t Robots Be Doing My Taxes By Now?
It's Tax Day, and if you're a software developer, I'll bet you find it as mystifying as I do. Not the actual tax preparation (mine are still pleasantly straightforward, I'm happy to say), but the general awfulness of the experience. Why am I responsible for collecting PDFs (or worse, paper) from a half-dozen institutions, then manually reentering that data? Why am I paying a vendor $50 for what amounts to some unit tests and an electronic transaction or two?
It makes no sense. Government uses technology for a lot of things, and some of those things are very hard [insert requisite reference to the Apollo Program here]. But filling out forms is not a hard thing. In fact, it's one of the problems that web technology has tackled first and most comprehensively. The first thing you learn in most web frameworks is how to make forms! It's hard to think of any other part of the government's mission that affects so many people negatively and could so easily and obviously be improved by better technology.
The IRS is trying to make progress on this score, of course. E-Filing has been with us since 1986. And they seem excited about the new version of their IRS2Go mobile app. But why on earth would I want a mobile app to help me find the IRS's YouTube channel?
Here's a better idea: instead of assuming I want to learn more about how to do my taxes, why not make it so that I can afford to know less about the process? Five minutes in a text editor tells me that my W-2 can be represented in less than 300 bytes -- a fraction of a QR code's capacity. How about promulgating some data standards that would make it easier for me to digitize all those 1099-INTs saying that I earned thirty cents on a checking account? Surely TurboTax or H&R Block would be willing to create some mobile apps that let me input my information by scanning a matrix barcode with my phone.
Better yet: since the agency is already receiving that data from all those financial institutions through a separate stream, how about organizing the data for me and simply letting me sign off on my automatically-generated return? I suspect that a lot of people would like that, given that the alternative is spending a spring day doing paperwork.
Naturally, this is not an original idea. As you'll see in these fine pieces from United Republic and the New York Times, many people feel that lobbying by firms like Intuit (the makers of TurboTax) has stopped efforts to make filing your taxes less unbearable.
Is this a case of malign influence peddling to prop up an industry that should be partially automated away, or is it just another example of government technology badly lagging behind that of the private sector? Whatever the case might be, here's hoping something changes soon. The fact that we're still doing our taxes this way is ridiculous.
Continue readingData for Better Bill Searching
I've put up a dataset on Github that maps popular search terms to bills in Congress. It's a simple, 5-column CSV designed to help people create better search engines that take in user input to search for bills. The idea is that this will be useful to, and get contributions from, the community of people out there working with legislation and building tools around them.
It's humble - I started it out with a mere 7 rows, assigning the keywords "Obamacare", "SOPA", "PIPA", and "PPACA" to the appropriate bills. There are certainly more good candidates than that, so please contribute via pull request, or if you don't know how to do that, open an issue and talk about it with words.
Continue readingThree Cheers for the CFPB
The financial watchdog agency announced an ambitious open source policy today, and we couldn't be more pleased at the news. The CFPB's announcement post does a great job of explaining their rationale: open source makes innovation easier, lock-in harder, and delivers value to taxpayers both by keeping procurements competitive and making sure their outcomes can be broadly shared.
It wasn't too long ago that government was scared of even using open source code, much less publishing its own. Its growing embrace by agencies like the CFPB and NASA is a testament to the hard work of organizations like Open Source for America. But it's also reflective of a long-established US norm that's only now being translated into the digital age: the federal government belongs to all of us. That's why our country's publications aren't copyrighted; it should be why its code is freely licensed, too.
At any rate, it goes without saying that Sunlight loves open source technology -- it's something we believe in and enjoy using. It's great to see that the CFPB feels the same way.
Continue readingLabs Update: April 2012
It's time again for our monthly Labs update. APRIL FOOLS! It's only been bimonthly so far this year! Oh, man, I got you so good.
Anyway, here's what we've been up to:
PyCon US 2012
Sunlight Foundation had a strong presence at PyCon consisting of Kaitlin, James, Thom, Paul and me. Our annual Open Government sprint didn't disappoint as we got a lot of participation on Open States and a new semi-announced project. This was also the first chance that Kaitlin and I got to meet Paul and Thom, confirming that James is in fact capable of hiring good people.
We also trekked through 20 minutes of office parks just to get to In-N-Out Burger. That's dedication.
Upwardly Mobile
We finally launched! Check us out on your desktop, phone or tablet at http://upwardly.us. We've also created a launcher app for Android devices so that you can have a convenient icon on your home screen.
Upwardly Mobile is Sunlight's first responsively designed site. We learned a lot in the process and hope to use this method more often in the future.
Upwardly Mobile lets you find out where in the country it's best to live by comparing various types of salary, living and employment data and ranking it based on your preferences. While economics are not the only factor in the decision of where to live, the app can be useful to help you find locations that you may have not considered before.
K3, IDEO
Work has begun on the third and final app, codenamed K3, for our grant with the Knight Foundation. We are taking a more thoughtful approach with this app, working with the fine folks at IDEO on the purpose and design of the project. Tom, Ali, Anu and I spent a few days out in San Francisco with the IDEO team getting a feel for their process and working through goals for the app. I also took the opportunity to eat massive amounts of raw oysters and fennel; my belly misses you, San Francisco.
We are entering a research phase now to identify the data sets that will support the app ideas that have been proposed. From there it will go back to IDEO for wire-framing and interaction design, then back to us for execution. Launch is scheduled for fall!
Check out our other apps from the Knight Grant:
Influence Explorer
Influence Explorer now contains frequently updated campaign finance data from the FEC! High five to Ethan for all of his work on this. The new data shows up-to-date fundraising for candidates and PACs, as well as new tables for Independent Expenditures. Alison worked on new data updates for campaign contributions, federal grants and contracts, and federal advisory committee data so that IE stays as accurate and relevant as possible.
Andrew has mostly been toiling away on Sunlight's upcoming regulatory-comments-tracking site, but also snuck off to speak at Transparency Works last week in Vilnius, Lithuania.
OpenStates
It's been a long time coming, but OpenStates has reached all 50 states, DC, and Puerto Rico in either production or experimental status! This is a huge accomplishment and a testament to James' dedication, talent and lack of anything else to do. Document-related features will be coming to the API, laying the groundwork for much-awaited full text search and document comparison.
Paul and Thom have been working hard on data quality improvements to move the remaining states from experimental to production status. The team will soon begin work on a new web interface for OpenStates, opening up the API data for anyone to browse.
Semi-public Scraper Project
Daniel put together a mini-site for the not-yet-quite-ready-for-the-public scraper project that uses ScraperWiki to collect congressional information. We ran some initial trials of the project at PyCon and a ScraperWiki event here in DC. We'll be making a decision soon if this project will graduate from the experimental stage. If so, we'll be sure to announce it here.
Team Journalistic Integrity
Jacob has expanded the super PAC tracker to include all independent expenditures and electioneering communications, as well as all committees making independent expenditures or electioneering communications.
Ryan's had the opportunity to report on two national news stories -- the Trayvon Martin shooting and the Virginia ultrasound bill -- using the SuperFastMatch technology that is being developed by Media Standards Trust. She used the tool to identify states that had adopted model legislation written by special interest groups. Ryan has also been working on stories and projects related to campaign finance in order to reveal who really influences the way Washington works and how they do it.
Lee has been working with Planet Money and This American Life on the value of congressional committee assignments and figuring out how much lobbying lowers corporate tax rates. Check out This American Life episode 461 that features the work of Lee and the rest of the Sunlight Foundation!
Team Leadership
Tom has been doing some traveling: San Francisco the K3 kick-off meeting with IDEO, and Austin to join a panel at SXSW and eat some tacos (mostly the taco part). In between he worked with Ryan on her SuperFastMatch project. At the moment he's working on hiring and a project proposal; next week it's back to the west coast.
Team Retribution
Ali has spent much of the past two months undoing an organization-wide re-branding she secretly undertook late last year. Ali was a huge fan of the CW's cheerleading show Hellcats and spent many months creating themed templates for all of our sites. You should have seen the theme for the Labs site: Alice, standing at the side with her injured wrist, giving the evil eye to Lewis and Marti as they practice. She'll have to find another outlet to mourn the cancellation of the show though; it doesn't quite fit into our mission as a government transparency organization.
This is a lie. Ali didn't send me anything for this update so I got to make it up for her.
Tidbits
- Tim just returned from China bearing gifts as consolation for any servers that broke during his absence.
- Poligraft is about to get faster and more accurate thanks to swell work from Dan.
- Capitol Words might be getting some new data sets soon. More on that in a future update.
- Eric has been working on getting a new government information search and alert web site ready for launch.
- Using technology developed from his compulsive need to track every item he purchases, Daniel has been working on a prototype of a bar code scanning app that would connect products to the political contributions of the companies that produce them.
- The final six sectors of SubsidyScope have launched!
- Our new designer, Amy Cesal, will be starting next week!
- We are still hiring.
- Album of the month: Threads by Now Now.
- Smoothie of the month: black cherry, strawberries, cocoa powder, and avocado from Yola.
Upwardly Mobile and Responsive Design
Upwardly Mobile lets you find out where in the country it's best to live by comparing various types of salary, living and employment data and ranking it based on your preferences.
After months of "coming soon" teasers, we are finally launching Upwardly Mobile! The site uses economic data provided by the government to help you find areas of the country where you, based on your occupation, could be more economically secure than where you currently live. While economic concerns are only one part of the decision of where to live, Upwardly Mobile can help you find areas of the country that you may have never considered before or help you rule out those you have.
Government: Do You Really Need An API?
As the term "API" has become more widely recognized through its ubiquity in social media and other web services, its coolness factor has grown considerably, and has become something frequently called for from government.
But does government really need to rush around and make APIs for all of their stuff? Peter Krantz argues that offering direct downloads to bulk data is a much more scalable, simple, and sane solution in most cases.
You should go read his article, rather than just our summary. But specifically...
Continue readingOpen States: 50!
Three years ago at PyCon 2009, we had the first PyCon Open Government Hackathon. Our big project was Open States (then the 50 State Project). The goal was to begin scraping state legislatures' websites in the hope of providing a common format for bill metadata across all 50 states.
Today, as we kick off the 4th Annual Open Government Hackathon at PyCon we're extraordinarily happy to announce one of the most significant milestones in the history of Open States: as of today, all 50 states (as well as DC and Puerto Rico) are now supported via our API and bulk downloads. This makes Open States the first and only completely open, completely free resource for accessing legislative information in a uniform format across all 50 states.
This is a proud day for all of us here and for everyone who has contributed to the project. Over the past three years Open States has grown to be much more than we'd envisioned and a great deal of that is due to great suggestions, contributions, and uses by the entire Open States Community. It is no coincidence that Open States has become Sunlight's most contributed-to open source project; we needed the community to make this project happen, and over 40 of you have answered the call.
Continue reading
59063
