It's back to work after a 4th of July filled with hamburgers, hot dogs, and other non-meat options. Here's what the Labs was up to over the past month...
Continue readingTo the cloud!
It's time that Sunlight Labs got its act together and joined the 21st century. Today we are proud to announce that we are partnering with cloud hosting provider Angelfire to move all of our sites to the cloud.
Of course, cloud computing offers concrete, well-defined benefits like agility, focus and flexibility. And cloud solutions can achieve things that traditional servers never could, like allowing users to leave their desks. Developers love the cloud, too: by now most are familiar with how simple it is to deploy to the cloud; many also appreciate how cloud datastores create more demand for developers. Plus, cloud solutions offer enhanced security both through obscurity, and technical monoculture.
This is a bold, forward looking move that we feel will help us accomplish our goals of transparency long into the future.
Continue readingDocument Freedom Day 2011
Happy Document Freedom Day! What are you doing to celebrate?
While it may not be obvious, open standards go hand-in-hand with open government. If we are asking our government to make information about itself available to the public, documents should be released in a way that everyone can access. Access to government information should not be only for the privileged few that can afford expensive commercial software needed to open files in proprietary formats. The public should be able to use any device, operating system, or software of their choosing without having to worry about what documents and data they won't be able to access.
Please note that while we advocate for the use of open document formats, it is more important to release data in appropriate formats. You won't get any praise from us for releasing earmark requests in ODF instead of a proprietary format. Structured data should always be released in programmer friendly formats such as JSON, CSV, and XML.
Additionally, open formats should be offered in addition to common proprietary formats instead of replacing them completely. We're not zealots here; proprietary document formats make sense for a large number of users. Just be kind to those of us that choose to not run the software needed to access the documents and give us an open choice.
Sunlight is celebrating by making the following pledges:
-
Any document created by Sunlight that is published in a proprietary format will also be published in an open document format.
-
We will update our recommendations to government to include the publishing of open document alternatives when proprietary formats are used. The Ten Principles for Opening Up Government Information hint at this, but we'll make it explicit.
So yay open document formats! Visit documentfreedom.org for more information or download this post in ODF. Sorry, I just had to do that.
Continue readingBlog Posts Via Email With CloudMailin.com
I recently learned (with horror) that a co-worker wrote her blog posts in Gmail, copied the rich text to WordPress, then copy and pasted the generated HTML into our Markdown-enabled blog backend. To be fair, our nerdy authoring tool is a bit much for non-technical users and doesn't really fit into most "normal" workflows. Additionally, she emails her posts to an internal list so Gmail was a natural authoring tool.
There had to be some common ground we could find; blog posts still written in Markdown while allowing her to use Gmail to write her posts. Our solution was to enable post-by-email on the blog. By adding a special email address to the recipients, the message is parsed into Markdown, a draft post is created, and she receives an email reply a few seconds later with a link to edit the new post. From there she can review and publish it in a few clicks resulting in a much improved workflow.
We wanted the draft posts created immediately and I didn't care to be polling a mail server every few seconds. Fortunately, we found a new service that made this project incredibly easy to implement.
CloudMailin.com
CloudMailin.com is a fantastic service that does the opposite of most other mail services. Rather than providing an API based method of sending email like Postmark, another fantastic service, CloudMailin.com receives email at a provided address and POSTs the data to a URL of your choosing. In addition to the simple parsing of SMTP headers and MIME parts, the service can handle email attachments. Pay them a few bucks extra and they'll upload the attachments to one of your S3 buckets!
A competing service we evaluated started at a pricey $30 a month; a bit ridiculous if we are receiving 5 emails a week to start. CloudMailin.com's recently announced pricing is right on the mark with a 200 message free plan and a 3000 message micro plan for $9 per month.
So how did we make it work? Let's look at some code...
django-cloudmailin
django-cloudmailin is a Django app we created to make working with CloudMailin.com as simple as possible. First we need a method that will receive the posted email message parameters and create a blog post.
In create_post we extract the parameters from the message to get the author, title, and content of the post. A post object is created and an email is sent back to the original sender of the email with a link to the Django admin for the new post. The author needs to check to make sure the post looks correct and hit publish. This is a greatly simplified example because we do some additional parsing of the content to transform the plain text into valid Markdown, but it should give you an idea of how it works.
Next we register that method with the mail handler.
MailHandler is a class-based view provided by django-cloundmailin that handles the registration and processing of mail messages. In this example we register our CloudMailin.com email address and secret key with the method that is to be invoked upon receipt of a new message. Multiple email addresses can be registered with the handler to allow for many different actions-by-mail in the same application. Finally the MailHandler instance is associated with an URL pattern in urls.py.
All incoming messages are signed with your secret key to prevent any old person from spamming your mail endpoint. The MailHandler instance takes care of verifying the signature so you can concentrate on writing your application.
You can find the source for django-cloudmailin on GitHub.
Continue readingdjango-mediasync 2.1 for Django 1.3
Earlier today we released django-mediasync 2.1 in anticipation of Django's upcoming 1.3 release. The Django 1.3 RC was released last night so the final version should be coming any day now. This release changes the way static files are handled and breaks previous versions of mediasync. The old MEDIA_URL and MEDIA_ROOT settings are now meant to handle media uploaded by users while two new settings, STATIC_URL and STATIC_ROOT, handle static site content.
Mediasync will first try to use STATIC_ settings and fall back to MEDIA_ if not found. This ensures that mediasync will work regardless of the version of Django being used.
Find the package on PyPI and the source on GitHub. And as always, if you use mediasync please indicate it on Django Packages.
Continue readingdjango-mediasync 2.0: Havana Nights
It's been almost a year since the last release of mediasync, but the new features we've worked on are worth the wait! If you use mediasync, please indicate that you do so on our Django Packages profile.
Source on GitHub: https://github.com/sunlightlabs/django-mediasync
Package on PyPI: http://pypi.python.org/pypi/django-mediasync/
Install with pip or easy_install:
pip install django-mediasync
easy_install django-mediasync
What is this media syncing you speak of?
For those of you new to the project, mediasync is a Django app that manages static media in both development and production. Imagine a project where you have to make updates to existing media, but all references are hardcoded to some absolute path in production. Do you update the production media and risk breaking the site or do you temporarily point to local media and hope you don't forget to revert the change?
With mediasync you don't need to worry about any of that. Paths to media are automatically generated: local in debug, remote in production, and manually overridden when needed. Modify your media in your local development environment then use mediasync to push the change to the remote production server. Reduce stress and add years to your life!
Continue readingSunlight’s (Mostly) Web-based Photo Booth
Sunlight Labs recently held an open house to bring members of the technology and transparency communities together over videogames and beer. Our systems administrator, Tim Ball, volunteered to create a photo booth for the event. A few days before the event Tim destroyed his arm in a terrible, unfortunate accident, nearly dashing our hopes for a photo booth. We had to honor Tim's memory (he's still alive) so rather than using an off-the-shelf photo booth software package, I hacked it up from scratch using Python, CSS3, WebSockets, and an iMac.
Continue readingOn DjangoCon 2010 and conferences in general
Sunlight Labs is a huge fan of Django. We use it in a majority of the projects we produce here and have released the source of numerous applications. So a few weeks ago a bunch of us eagerly packed our bags and flew out to Portland, OR for DjangoCon 2010.
Continue readingSection 508 compliance is still easier than you think
This is part two of a two-part post. Part one covers the basics of web standards and progressive enhancement and Section 508 standards §1194.22 (a)-(f). Part two covers Section 508 standards §1194.22 (g)-(p).
Good news! Despite the excessive amount of time it took me to finish this post, Section 508 compliance is STILL easier than you think. Compliance does not preclude you from having an amazing web site. By following modern web standards, it is possible to create a site that is inherently accessible. Let's continue where we left off with (g)!
Continue readingData Commons Matchbox
Earlier this year we started on the Data Commons, a project to merge open government data sets to make them more searchable and usable. Our goal for the initial release is to load state and federal campaign contribution data from The Center for Responsive Politics and The National Institute for Money in State Politics. Along with the raw transactional records, we will be taking the additional step of matching the entities (people, organizations, corporations, etc.) across the data sets. We'll have more posts soon with details about the Data Commons.
To assist us in this effort, we are developing Matchbox, a toolkit for the merging and matching of entities. We have big plans for Matchbox, but want to get feedback from the community as we improve it over the next few months.
Continue reading