With the dust finally settling from TransparencyCamp, it’s a good time to talk about some of the work we did this time around to prepare for one of our favorite weekends of the year. TCamp ’13 was a year of firsts for us: our first 500+ crowd, first time running the event with a mostly volunteer staff, and our first time trying to run nearly everything digitally and online. In this post I’ll run through the what/why/how of that last point, and touch on some lessons learned. It’s gonna be long and Django-centric, so grab your pink pony mug and cozy up with a cup of coffee.

Embracing Constraints

The TCamp site historically has run on a two-year refresh cycle, so we knew well ahead of time that we wanted a sizable update for this year. However, not everything we built was purely of our own volition. Two major components–the pre-camp brainstorm and the Wall–had to be reconceived for reasons beyond our control.

The Brainstorm Site

In October 2012, Google deprecated the Moderator API. In previous years, Moderator was where we ran our brainstorm, and it generally provided a lot of value for very little effort. Sure, an account with Google was a prerequisite for participation–but that aside, a deprecated API meant getting content out of moderator was going to be a data entry task, and, well, I couldn’t find anyone who wanted to do it, so rolling our own was looking pretty attractive.

The TCamp13 Brainstorm Site.

We love Django at Sunlight, and with 221 public repos hanging around our github page, chances are if you need a django app suited to a particular task, @jcarbaugh or @jamesturk has built it at least once already. Such was the case for a basic voting app, and django-brainstorm ended up providing at least some moral and spiritual guidance for what would become the TCamp13 brainstorm site.

Before you run off and install django-brainstorm to your own apps, though, be aware that it’s collected quite a bit of dust in the three years since the last commit. I ended up rewriting a good chunk of it for Django 1.5, and will be either merging my work back in or extracting it as a new component soon. Some major differences aside from CBGVs include python-social-auth for logging user votes, configurable labels and html snippets for much of the domain-y stuff (you can call it an Idea, Post or Proposal, a vote or a +1, etc.), Disqus comments, and at least the basis for pluggable scoring algorithms, though none other than total score has been built out. We adapted our implementation for TCamp to display scoring bars similar to how our Moderator page looked in years past, and at the end of the day had an app that was identical in functionality to what we’d had before, but now with a custom look and feel and one-click conversion of winning ideas into proper sessions, ready for the Wall.

The Wall

Ah, the Wall. The crazy whirlwind of typing up hundreds of 2012 session proposals was still fresh on the minds of several of us who’d served in the Wall-setting trenches last year, and we weren’t super eager to relive it. When we were strongly encouraged by GW event staff to help keep the paint *on* their walls by not attaching anything to them, we decided to weigh our options for going digital. We rented a short-throw projector and took it to the venue to get a brightness level, and convinced of both the viability and outright awesomeness of the technology (you can get a 6-foot-high image from just a couple feet away!), placed an order for 3 of them to be run side-by-side as our Wall, at a total resolution of 3,840 x 800.

The Wall, in action. Photo by ebelisa.

While on its surface, the idea of projecting a big HTML table stretched across three screens seems simple enough, the reality of managing a 13-track, hour-by-hour calendar of events on the fly is astoundingly complex. For starters, we assumed we’d have a single session model that would drive the Wall display, the schedule on the website, the TV outside the rooms on the upper floor, and a new SMS query service. We also set out to have each schedule block automatically tweeted 10 minutes before it started, though we didn’t end up using it. We decided that campers should be able to submit a session from their laptop or mobile device, and they should be able to edit it in case they made a mistake, up until the time it’s public and on the wall. Perhaps most importantly, Campers submitting sessions should get email notifications about the status of their session, so they know where and when they need to show up to lead it. In order to help campers focus on where to go next, past events on the schedule should be hidden out of the way, leaving only current and future programming in view.

The various modes the schedule took on for TCamp13.

That’s a lot of specs, and we haven’t even considered how scheduling will work on the back end! At a scale of 500 campers leading nearly 130 sessions over two days, it was clear that the admin interface needed to handle as much of the load as possible. The admin site is one of Django’s finest features, and fortunately it wasn’t too hard to put together a streamlined system for scheduling lots of events all in one go. We used list_editable fields to make the start time and location editable from the django change list, meaning that many unscheduled events could be assigned all at once and saved with one click. End times were determined on save by a configurable session_length attribute on the event model, meaning only the start time had to be entered. Locations were pre-populated into a related table, and came up via django-grappelli‘s autocomplete feature. Events for an entire timeslot or day could be assigned rooms and start times, and then published all in one go using custom admin actions.

Editing the wall at TCamp13

Finally and completely aside from the logistics of just placing events in timeslots, it’s a huge challenge when scheduling 13 parallel conversations around the general subject of “transparency” to reduce duplication of topics and to help enable people to make it to as much of the stuff they’re interested in as possible. At this year’s camp every talk was a great conversation led and attended by amazing people, so this was of course a failed proposition from the start. Still, we wanted to do our best to categorize all the incoming content and stagger it appropriately. To this end, we asked session submitters to provide a few keywords that summed up their idea, and used django-taggit to manage them. At the same time, when each talk was submitted we ran it through a tag extractor library to automatically pull out high-value words from the title and abstract. This helped make sure that there was some approximation of the content even when user tags weren’t provided. We displayed the tags in the change list to help schedulers quickly avoid timing conflicts between similar talks. Ultimately, the best feature of the scheduling process was the folks managing it. @boblannon and @caitlinweber dutifully and adeptly considered each talk and did an awesome job juggling the needs of campers with the imperative to fill all of the rooms and make sure everyone who wanted to got a chance to lead a session.

The wall team, hard at work.

No Deployment on Game Day!

Another hazy (mostly repressed) memory from TCamp 12 was having to push code out to the server from an overloaded, sporadic wifi connection in order to keep things running smoothly. It was a personal goal of mine this year to address each day-of need that I could possibly anticipate in such a way that it could be handled from the django admin–even from a phone on a cellular connection if necessary. As a result, all of the content on the site, from the navigation to the static pages to the sponsor info is managed via django admin. The page system in particular is pretty cool, and it’s also prominently used on the foundation site. The general idea is similar to the django flatpages app, only with the goal that nothing is really flat at all. Each page is a renderable django template, with access to tag libraries, context variables and blocks. Pages can extend existing templates available to the installed template loaders, or they can provide templates in the database using its own included loader–we generally don’t make a habit of doing this, but sometimes it comes in handy for quick one-offs. Pages have a reverse foreign key relationship to template blocks, which can be added arbitrarily to override any template region. At render time, the content area and any attached blocks are concatenated into a template string, which is rendered against the current RequestContext. That template string can be defined in settings, but looks like this by default:

DEFAULT_BASE_TEMPLATE = u'''
{{%% extends "{template}" %%}}
{{%% block title %%}}{title} - {{{{ block.super }}}}{{%% endblock %%}}
{blocks}
{{%% block content %%}}{content}{{%% endblock %%}}
'''

Even better, content and block regions are markupfields, so they can be written in html, markdown, or whatever format suits the author. This has proven to be a great way for us to quickly build powerful static pages that integrate seamlessly with our existing site templates, and TCamp was no exception. Pages here were also hooked into django-reversion to make sure that irreplaceable database content couldn’t get lost by accident.

SMS, Just For Fun

We love Twilio at Sunlight and have built a few things over the years making use of their great APIs, not the least of which help provide information to low-bandwidth users or those who don’t have access to a smartphone or computer. We thought that should carry over to TCamp as well, not only as a backup in case the wifi completely failed on us, but also for the speed and simplicity of texting. With that in mind, we put together an SMS interface using Twilio to respond to queries like ‘now,’ ‘next,’ ‘lunch’ or ‘1:30pm,’ responding with information about what sessions were scheduled, and where, or what time the food trucks were arriving. Django-twilio made this a breeze, and thanks to some handy model managers doing the the retrieval lift, the entire interface is less than 70 lines long.

Build on Heroku, Deploy Anywhere

I’ve gotten in the habit of building all new projects against a heroku staging environment. Heroku has proven to be a great baseline for portability, from ensuring an automated build process can bootstrap your app just from the source repo to dealing with static files and user media externally via S3, to understanding how your project will run on low-powered hardware, Heroku helps by providing some sound assumptions about your architecture. One drawback, though, has always been local settings. Heroku uses environment variables to manage settings, and this conflicts a little bit with the way most django apps work–importing a local_settings.py file from the local file system. To ease some of this pain I created Herokal Settings, a django app that takes care of toggling between Foreman/environment variables, or your local_settings.py. It also provides an exportenv command to dump your local settings into a .env file for pushing up your configs to Heroku in one step.

Lessons and Takeaways

While we were more prepared than ever this year, there are always pain points. The big lessons we’ll be carrying with us into planning for next year are:

• Even at a pretty technical event, most attendees are going to leave their laptops at home. Treat mobile as the primary day-of experience. This seems obvious, but anecdotally I was surprised by how many fewer laptops I saw this year than in 2012, even though the crowd was bigger. More and more folks asked for lightning connectors to hook their tablets up to project their slides, and more people submitted talks from their phones than from computers, either their own or the couple we had available. We designed a responsive site that cut out a good chunk of imagery and video on mobile devices, but that turned out to not be nearly enough. The resulting mobile experience had way too much going on for lower-powered devices–a dedicated mobile site consuming the API would have been a better choice.
• Scheduling sessions is a single-threaded process, no matter how much tech you throw at it. While moving session scheduling online created a lot of opportunity for extra value, it didn’t speed the process up as much as we expected it to. At the end of the day, everyone scheduling talks is pulling from the same pool of proposals and it goes much more smoothly if it all happens in one place. Perhaps we could have made it work collaboratively with websockets and some really well done UI, but it wasn’t a terrible experience setting the wall from one machine, and certainly better than it would have been from two.
• Always have a plan B. Even though we didn’t need it, we made sure that if things went horribly wrong we could print all of the sessions out on paper and tape them up in brazen defiance of venue policy. It was comforting to know this fallback was there, and anyone attempting to cut the analog bits out of an unconference should have them available too.
• Push all of the schedule information as far forward as possible. Some of the most consistent wall feedback we got was that while it looked cool, It was hard get a good sense of what each talk was about from the 50-ish character title that was displayed in the grid. Due to limited screen real estate, we shifted this year from the single ‘abstract’ field written on a piece of paper in years past to a short title and deeper description, the latter of which was present only on the website. Couple this with a slightly-too-heavy mobile experience and not a lot of laptops in the building, you can easily see how this was suboptimal. I’m still not sure how best to solve this one, as it’s tough to find projectors that do more than 800 vertical pixels, but you can be sure we’ll be giving it some good thought for next year.
• Collaborative notetaking should be distributed!We love etherpad, and have used it for notes at the last two TCamps. However, its technical implementation isn’t all that well-suited for several hundred users on a single instance. This year we struggled right out of the gate to keep our etherpad instance up and running. First, we found that we’d inadvertently put it behind our varnish box. We typically route through varnish even when we’re not caching with it for the app server portability a fast proxy affords. However, etherpad’s websocket / long-polling architecture is a terrible fit for varnish, and we started the morning with just about all of our sites offline due to a varnish box swamped with waiting connections, which was quite perplexing until we figured out what was happening. After DNS resolved, things were looking a little better, but one instance still couldn’t handle all of the traffic. If we had it to do again with etherpad’s codebase held constant, we’d create an instance for each room, sharding the load while maintaining deterministic urls that can still be programmatically embedded into django templates.

That’s It!

Thanks for reading, I hope this shed some light on some of the more technical aspects of our planning process, and here’s looking forward to TCamp ’14!