It’s the most wonderful time of year… Montgomery County property tax payment time! It’s also the holidays, which are quite nice as well. Things are wrapping up here in the Labs before we head off for winter break. We have a lot going on right now and even more big plans for next year.
In tangentially related news, Scott Weiland released a holiday album. I can sense your blank stare from here… please don’t let it distract you from reading the rest of this post.
Influence Explorer
The Data Commons team has launched a redesign of Influence Explorer that greatly improves navigation on long, complex profile pages. As you scroll, the navigation bars stay with you so that you know which data set you are currently viewing and can jump between them quickly. The year selector also follows you so that you can easily switch to different year views.
Ryan and Lee have been working closely with Ethan to dig through the data stored in Influence Explorer. Interested in reading up on lobbyist bundling for the Super Committee? How about the political ties behind Zuccotti Park? Want to find out how lobbying can reduce your tax rate?
In addition to all this lovely work, the team has been acquiring more timely campaign contribution data from the FEC, exploring the federal regulatory process and upgrading the server infrastructure.
Open States Project
James and contributors have been knocking out the states, bringing us ever closer to 50 + DC. Kentucky, Oregon, Idaho, Arkansas and Nevada have all graduated from experimental status based on several months of stability. North Dakota and South Carolina were also recently added to the API.
James has been prepping the Boston Sunlight office, new home of the Open States Project. He just hired a new developer and has secured office space. I patiently await an invitation to the opening party.
Congress for Android
Eric released a major update to the Congress App for Android that includes a visual redesign and information on what’s coming up in the next couple of days on the floor of Congress. This is a really great release and Eric did a lot of great work on the new redesign. He’s got many plans for new features that will be included over the next year, so stayed tuned!
The section in which I post Chris' update verbatim
Chris wishes that there was a more eloquent and loquacious manner in which she could describe her continued work in the mobile game app and the 180 Project. Alas, these projects defy description as the day-to-day minutia of design eventually amounts to: move this there, rinse repeat. However, Chris is pleased to report that the completion of the 180 Project is in her sights, barring any timeline disrupting events. She is coding, thus all is well.
Team Sysadmin
Tim has been involved in the long and arduous process of upgrading our office network. As it currently stands, the new fiber connection is a frustrating 15 feet from the office. Tim can see it from the ceiling tiles above our server room, but it is caught up in insurance, contractor and building management turmoil. To ease his mind, he’s been configuring our new Juniper Junos EX-series switches. It’ll be like a cute little ISP here in Sunlight’s office!
Team C-Level Executive
Tom is freshly back from the TAI Bridging Session and News Foo. Aside from that he’s been working on filling our open positions and some end-of-year planning stuff.
Tidbits
- Expanding on the Sunlight Labs Olympics, we’ll be participating in the Sunlight Foundation Olympics early next week. Results will be posted shortly thereafter!
- We now have 40 instances running on Amazon EC2. I’m sure we know what’s on each of those boxes, right?
- Drew has been lending a hand to reporting to keep their projects running while we search for someone to fill the open position.
- Dan and Capitol Words. Soon. Promise.
- Eric and Andrew begun a project on gathering the data to connect bills and laws to the regulatory process. This effort should yield lots of bulk data over the next month or two for the legal and legislative communities to use.
- Kaitlin has updated the video endpoint in the Real Time Congress API to support some upcoming changes to our Roku apps.
- Upwardly Mobile is coming together nicely. There will be lots of great things to show early in January.
- Renaissance man Luigi Montanez authored How can software engineers help make government better? in the latest issue of the ACM’s XRDS (Crossroads) magazine.
- The hottest Labs holiday gift this season is Well Dressed’s El Gordo burrito.
When working with raw meat for your holiday meals, remember: though Sunlight is said to be the best of disinfectants, bleach is better.
Continue readingLabs Update: November 2011
With a regularity typically seen only in cron jobs, the monthly Labs Update is coming at cha! As you read this post and plan your Thanksgiving dinner, listen to Thanksgiving Theme by holiday season favorite, Vince Guaraldi.
Halloween Open House
If you didn't make it out to our Halloween Open House, then you missed out, my friend! Between the dry ice, Chocolate City Beer, taquitos, halloween costumes, photo booth and demonstrations of a colloidal suspension, we actually got a few chances to talk about the work we do here!
The best part, though, was Eric's Kinect and Processing powered rendition of Vigo the Carpathian. You can find the source code on GitHub.
Goodbye and Hello
We recently bid fond farewell to our reporting team embed, Aaron, who has taken a job with the Huffington Post. His position is still open, if you are interested in applying.
Joining us in the Labs is Lee Drutman. He will be working as our data visualization fellow, exploring interesting uses of the data with which we work.
Six Degrees of Corporations
Ever had to identify a unique corporation in a large government data set? How about corporate subsidiaries? You know that it can be almost impossible, but our new project, Six Degrees, shows you just how actually impossible it can be. The project looks at DUNS numbers used in federal spending data and finds that inaccuracies in reporting, among other issues, make the identifiers virtually useless.
Kaitlin and Drew created Six Degrees from their work on Subsidyscope. My description isn't doing the project any justice so go check it out for yourself!
Capitol Words
Soooooo close! A few weeks ago we gave an internal preview of Capitol Words to the rest of the Sunlight staff. The feedback was so good that we felt it had to be incorporated before the official launch. Caitlin and Dan have been working hard on the new features and should be finished within the next week or two with a launch soon after that.
Upwardly Mobile
Formerly named Moving Up, Upwardly Mobile is the second in our series of Knight Foundation funded mobile applications. Having a tough time making ends meet? Upwardly Mobile helps you find other ares of the country where someone in your occupation can make a better living. We're pulling in a number of government data sets that will be fully explained as we near launch. Caitlin is just wrapping up Sunlight's first responsive design for the project and it is looking amazing. I'll be templateing and making it all work over the next few weeks. Look for a blog post later this month about the progress!
Influence Explorer
With election season upon us, the Influence Explorer team has been working on tools for tracking candidates and contributions. We've often pointed out the problems with less-than-timely release of data, especially campaign finance data. The issue is painfully evident during elections when clean data can be many months behind. You go into the voting booth without knowing what has happened in the previous weeks or months of the campaigns. We're limited by the FEC release schedule, but there is a plan in place to get some information to you as soon as we can. Ethan should have more details for you soon.
In addition to these new features, Alison has added a new data set, Lobbyist Bundled Contributions, and has refreshed EPA and federal campaign finance data.
SunlightFoundation.com
We're embarking on a complete refresh of the Sunlight Foundation brand! Along with a new logo and updated visual design, we are rethinking the organization of our content. The new SunlightFoundation.com will be organized around the work we do and not our internal team structures. We'll be pulling in some of our department specific sites like labs and reporting back under the Sunlight Foundation umbrella.
Ali recently blogged about the design process and gave a sneak preview of our new logo. Meanwhile, I've been whining about having to setup WordPress and have been trying to find ways to weasel out of it. We'll be blogging more about the progress and hope to launch late 2011 or early 2012.
Sunlight Live and Datajam
October saw the first Sunlight Live produced on the new Datajam platform. The card system has been completely rewritten in both the front-end and the admin. While we've so far reused the existing 3rd party widgets, Dan and Luigi are continuing to implement our own versions that will make Sunlight Live even better than it currently is.
Team Executive Leadership
Tom has been spending a lot of time on grant reporting, proposal documents and making sure things are in motion for staffing back up in DC and Boston. He also briefly embarrassed himself by attempting to close some tickets on Capitol Words (this was a mostly terrible idea). And he spent some time out in SF for the Code for America Summit, which was both fun and inspiring.
Tidbits
- Chris has been busy working on design for the 180° Project.
- Daniel's been plugging away at an analytics dashboard that will be used to provide an at-a-glance overview of various types of traffic and impact analytics.
- Minor updates were released for our native mobile apps, Congress for Android and Real Time Congress for iOS.
- Tim now has graphs of everything our servers are doing. He has also been shutting down old and unused servers, much to our checkbook's delight.
- Kaitlin attended Open Government Data Camp in Warsaw, Poland.
Official Sunlight Labs Thanksgiving side dish: Oyster stuffing.
Continue readingOn content management systems and an unreasonable need to DIY
TL;DR I want to write a new CMS and force people to use Markdown. Should I?
As Ali mentioned in a previous post, we are embarking on an effort to redesign and reorganize SunlightFoundation.com. Part of the reorg involves consolidating our "brands" back under the Sunlight Foundation umbrella so that our content reflects what we do and not our organizational structure. The plan is to merge our existing labs and reporting blogs into technology and reporting channels on our main blog.
With this, though, comes some major technical challenges. One such challenge is having to merge content from a myriad of existing CMSs. We have one blog running on our own django-blogdor application, another running a forked version of django-blogdor (don't ask), and two more blogs running WordPress on the backend but using django-wordpress for the public facing Django-based site. Once the blogs are dumped into an Atom-based format (with custom elements for additional metadata) they can be reimported into the new content management system. What that system will be is a decision that causes me constant angst.
We could just use WordPress like we are now where a private instance is used for authoring and Django pulls from the WP database for presentation on the public site. There are some requirements that would require us to develop WordPress plugins to keep track of additional metadata and make an internal version of django-wordpress that is aware of the plugin tables. To make things even more complicated, there isn't a good way to map Django models to WP tag/taxonomy tables since Django models cannot handle compound keys. This results in quite a bit of database overhead when tags on a large number of posts are accessed. Another option would be to take this a step further and create a generic RESTful API around the WP database that our public site can use. This approach, which would allow us to swap WordPress out for other blog engines at will, is being used by Talking Points Memo.
But if we have to do all of this development just to use WordPress as a backend content store, why use it at all? Why shouldn't we just reinvent the wheel and write our own CMS? The actual management of content isn't hard; django-blogdor can do this just fine and would require only minimal improvement. The hard part, which WordPress does well, is the authoring interface.
Ah, the authoring interface. That brings us to the biggest source of contention. Whenever I discuss this project with the rest of the organization, I'm always told that a WYSIWYG editor is crucial. While I understand the need to insert media and add headings, I'm less sympathetic to other forms of visual styling such as the changing of font colors. I find myself increasingly convinced that content creators should be using Markdown to author their posts in a tool that has WYSIWYG-like helpers for inserting chunks of markup or HTML. Not only does this produce cleaner HTML that is more fault tolerant to future changes, but it creates a clear separation between the creation of content and it's visual display. Plus Markdown is just easy; you can pick it up in a few minutes. More complicated visual needs can and should be handled by our design team or built into the CMS. I want to free our authors from having to worry about presentation and focus on what they do best, writing good content. How considerate of me!
So I come to you, dear reader, for advice on what we should do.
- Am I being unreasonable here with my urge to write yet-another-CMS? Feel free to tell me to shut up, suck it up, and make it work with WordPress.
- Is it snobbish and elitist to expect content creators to use either Markdown or learn HTML rather than use a WYSIWYG editor?
- Any better suggestions?
PyCodeConf 2011: Snakes in an Infinity Pool
If there is one thing that I learned from PyCodeConf, it's that all conferences should be in Miami in October. And they should all feature parties at rooftop infinity pools. Aside from the fun, PyCodeConf had a great selection of speakers that showed the breadth of the Python community, from wedding web sites to scientific computing. Read on for an overview of the some of the talks that pulled at my heartstrings.
The slides and audio from all talks can be downloaded from the PyCodeConf site.
What makes Python AWESOME?
This talk by Python core developer Raymond Hettinger was one of my favorites. When working with a language on a day-to-day basis, it is easy to take features for granted. Iterators, generators, and comprehensions are things that seem simple at first, but allow you to do very complex operations in very little code. The new-ish with statement provides an elegant interface for resource management and separation of common set up and tear down code.
Physics is renowned for the beauty and elegance of it's theories and equations. It's these same principles that made me love Python. While the language is slower in gaining new features, you can be guaranteed that the implementation will be incredibly clean and consistent with the principles of the language.
Embracing the GIL
I was fortunate to see David Beazley give a GIL thrashing talk at PyCon and this talk was just as good. The GIL is a very controversial part of Python which has both FUD and actual issues surrounding it. David has done a lot of research into how the GIL works and demonstrates how it behaves under various conditions. The summary: Python 2.7 is okay, Python 3 needs work, and a basic implementation of thread priorities in Python 3 puts it on par with 2.7.
API Design and Pragmatic Python
Kenneth Reitz is best known for his wonderful packages such as requests, envoy, tablib, and clint. If you've used any of Kenneth's projects you'll have noticed that he values creating sensible APIs that insulate users from the messier parts of Python. He takes a very conservative approach to his cause; no need to actually replace messy packages, just create wrappers that make them easier to use.
Kenneth also announced the release of The Hitchhiker’s Guide to Python. His goal is to create a central repository for Python best practices covering everything from installation and editors to coding style and app layout.
PyPy
The one common theme of nearly everything at the conference was PyPy, famed alterna-interpreter. The team has come a long way and everyone was eager to show the areas in which it excels over CPython and point out the parts that need some work.
The general consensus seems to be that over the next few years PyPy will become the interpreter of choice for running Python. The team is currently accepting donations on their site for general development, Python 3 support, and a port of NumPY. I've donated, you should too!
Who's coming with me next year?
I highly recommend checking out out each of the talks. Even though I only highlighted a few here, they were all quite excellent. Thanks to GitHub for putting on such a great conference and all of the sponsors that allowed it to happen (free mojitos).
Continue readingLabs Olympics: Rex the Cleanosaur
We've got a problem here at Sunlight. A tiny kitchen and a large staff is a recipe for a disaster of immense magnitude. Each day a member of staff is assigned to kitchen duty; the list on the fridge has a schedule of who is responsible for keeping the dishwasher full and run as needed, wiping down the counters and making sure that dishes are put in the appropriate cabinets. Some staff members do a really great job, others are inconsiderate assholes who let the kitchen go to pot (I admit to being one of the inconsiderate assholes on occasion). We're smart people, there has to be a better way to do this!
Rex the Cleanosaur is a better way.
For the 2011 Sunlight Labs Olympics, Team Awesome created Rex the Cleanosaur, a kitchen duty management application. Rex generates a schedule from current staff members and emails them with their assignments. If you receive an assignment for a day that you will be unavailable, just click the link provided in the assignment email to defer your duty. Another staff member will be automatically scheduled to take your day, but the next day they have kitchen duty, you'll do theirs. Just to keep everything fair, the web app makes deferments publicly available. Rex will chase down and eat habitual deferrers!
We also developed a tablet based interface that will be hung in the kitchen to display the person that has the current kitchen duty. But what if that person is doing a terrible job? How awkward it would be to have to talk to them face-to-face to ask them to do a better job. No worries! The table interface has a "nudge" button that, when tapped, sends the person responsible for the kitchen an anonymous, passive aggressive email telling them to get their act together. If they are doing a great job, you can "throw 'em a bone" to thank them for their excellent work.
The nudges and bones are used to calculate rankings of the best and worst kitchen duty. If the person responsible for kitchen duty has a high nudge count, you know that they will probably need some extra reminders throughout the day. Managers could even use nudge/bone counts as a factor in determining yearly raises! Okay, not really, but you can still berate people with high nudge counts for their lack of consideration for their fellow employees.
So what's next? We plan to add a way for the person with kitchen duty to rate the office as a whole on how well they held up their end of kitchen cleanliness (placing dirty dished in the sink rather than the dishwasher, etc). There are also some early plans for coupons, exemptions from kitchen duty that managers can give away as rewards for good work or to pay off employees that stumble upon secret evil plans their managers are working on.
Team Awesome consists of Chris Rogers, Drew Vogel, and Jeremy Carbaugh.
Continue readingLabs Update: October 2011
We've had the first hint of fall here in DC with Metro switching from AC to heat. Grab some apple cider and a baked good while you read this brisk update from Sunlight Labs.
Eric Mill, International Man of Transparency
For the last two weeks, Eric has been in Santiago, Chile, working in the offices of the Fundacion Ciudadano Inteligente. There, he's been chipping in on an API over Chilean parliamentary data, imparting what knowledge of JavaScript and Android he can and learning about the open government scene in Chile.
Influence Explorer
Yet again, new data sets have been added to Influence Explorer and Transparency Data. This past week was the official launch of EPA violations, public comments on federal regulations and corporate employees that sit on federal advisory committees. The BP profile page is a great example of each of these new data sets. Andrew has also spent some time working on infrastructure improvements to allow us to keep our copy of the Regulations.gov data up to date.
Real Time Congress
Before his international travel, Eric worked on adding Senate Floor information to both the Real Time Congress API and to the Congress app for Android and continued development on an upcoming website around alerts of government data. Being the spontaneous person that he is, he made a spur-of-the-moment trip to the Senate Office of Public Records to fetch documents relating to Senate Political Fund Designees, which he'll digitize and release as a CSV after returning from Chile.
The Open States Project
The Open States team has continued to expand their coverage with the addition of Nebraska and Iowa. The team is also working to add more special sessions and several features utilizing the actual text of legislation. James will soon be heading up north to Boston so be on the lookout in the coming months for exciting Sunlight North news.
Sunlight Live / Datajam
Luigi has continued to plug away at Datajam, a new platform to power Sunlight Live. In addition to the core platform, Dan has been working on a modular comment and discussion system for events. If you've been hankering to run your own real-time, data-driven events, you won't have to wait much longer. Datajam will be well-documented and open-sourced for everyone to use by the end of the year.
Subsidyscope
Clearspending 2011 has launched! In addition to the launch, Kaitlin and Drew have been working on a corporate identifier project. The site will feature a visualization of the impracticality of continuing to use DUNS numbers as the de facto standard for identifying corporations. It is based on a 2D JavaScript port of the Traer processing library and the "nice branching" processing sketch by Natasha Harper and Katie Adee. Kaitlin is currently working on two presentations for the Association for Budgeting and Financial Management conference (super wonky) and one for Open Government Data Camp in Warsaw (super European).
K2
K2, the codename for our second Knight Foundation mobile app, is moving along nicely. The final name and logo are days away from approval, data has been finalized and scraped, and design is moving from comps to build out. Ryan, Caitlin, and I are quite excited about this app and hope to have something good to show you for next month's Labs Update.
SunlightFoundation.com
The big news on the Foundation front is that we are embarking on a rebranding, redesign, and content reorganization of the main SunlightFoundation.com web site. Ali is getting to work on comps and mood boards and I'll be looking at a reorganization of our blog content. In addition to the redesign, we've been pushing out new campaign pages. See our Super Committee page to find out what we've been working on to bring transparency to the committee.
Team Design
Now back to full staff, Team Design has been swamped with new work. Chris has been knocking out presentations for Sunlight speaking engagements and worked on design for a few new projects. Caitlin has been working on Capitol Words (I promise, it's coming soon!) and K2. Ali has been working on the design for Kaitlin and Drew's corporate identifier site, a new visualization for federal budget projections over time, updates to Influence Explorer, the redesign for Sunlight Foundation and a microsite for the Reporting Group for research they're doing on financial databases.
Team Management
Tom has been doing the usual odds and ends, wrapping up some grant proposals and writing some reports for grants we've been working on. It's all very glamorous! Somewhat less boringly, he's also getting ready to head to the Code for America Summit next week. The big news, however, was the launch of Superfastmatch, a joint project with Media Standards Trust. Superfastmatch is an open source tool for quickly finding overlapping text between documents in a huge corpora. We've got some great stuff planned for Superfastmatch so stay tuned.
Team Sysadmin
Between searching for an entrepreneur to marry and making scallion oil pancakes, Tim has been working to improve our core infrastructure. In the office Tim has been buying new hardware to replace the switches we've had since we were a very small organization. Once the new hardware is installed and the chains of wireless routers are removed, we hope to have dead-zone free wireless access throughout the office. We now have data-filled Munin graphs for all of our servers. Tim also went to a Varnish meet-up at the SURGE conference.
Tidbits
- Aaron has been helping our reporting group make improvements to their site that will allow users to keep up with electronic FEC filings, particularly those from presidential candidates and members of the Super Committee.
- Our mobile game, Decipher DC, is on hold for the moment as Daniel works on an analytics and metrics dashboard.
- I'm headed to sunny Miami, FL tomorrow for PyCodeConf. Send me a message if you are going so we can meet up!
- The Labs is holding an open house later this month, so keep an eye on the blog here for information. We'll post invites once we figure out the maximum number of lasers and dry ice we can buy.
- Goodbye, 1800 Cafe! You've fed us well with delicious scrapple, egg and cheese sandwiches since we've been here in Dupont Circle.
Labs Update: September 2011
Summer vacation is over and it's back to school. Read on to see what we've been up to over the past month, including irradiating babies with thousands of IR beams!
Continue readingDevelopers and federal agencies: Can we talk?
I recently posted about the good work the EPA is doing to get the developer community involved in their challege, Apps for the Environment. As part of their effort, the agency has been hosting a series of webinars where EPA experts discuss the data and tools that are being provided for the challenge.
This Thursday at 4pm I will be appearing on one of the webinars discussing the role of government challenges and their potential to open up agency data and spur innovation. I'll focus on what agencies can do to provide developers with the information they need and ensure that the applications that are produced are worthwhile and self-sustaining. Joining me will be:
- Alex Howard of O'Reilly Media, moderator
- Michaela Hackner of Forum One
Registration for the webinar is still open.
Continue readingLabs Update: August 2011
Barely survived the hottest July ever in DC? That was nothing compared to this sweltering wave of updates from the Labs. While you read, cool yourself off with the song that topped the Billboard Top 100 chart this day in 1991...
Continue readingEPA’s Apps for the Environment Challenge
App contests are a dime a dozen these days. Right now there are 111 listed on challenge.gov. A quick review of the challenges showed that few agencies do anything to reach out to developer communities once the contest is posted. The EPA, however, has done a fantastic job with their Apps for the Environment contest by actively working with participants. Ethan McMahon, dedicated public servant and EPA employee, came to the last django-district meetup to try and get developers involved in the contest.
The EPA has compiled a great list of data sets and web services for developers to use. If you've looked these over and still don't have an idea, there is a curated list of potential app ideas to get you going. Still without an idea? The public has been submitting their own suggestions on the EPA's data blog.
One of the biggest issues with app contests like this has less to do with the quality of submitted applications and more with the proper usage of the data; many of the datasets that the government produces are quite complex. People have spent their entire careers becoming experts in their field, so how can a developer understand the data in the few months they have to create an app? EPA has really excelled in this area by hosting weekly webinars (I hate that word, it's netposium from here on out) to help developers understand the data sets they are working with. You can suggest a netposium or ask specific questions about the data by emailing greenapps@epa.gov. I'd love to see the netposiums continue after the contest ends so that anyone working with EPA data will have a rich resource to which they can refer.
Submissions are due by September 16, so get coding!
Continue reading
59063
