Unlike last year, I wasn't just a mere spectator of the Labs Olympics. I got to participate this year and take a couple days off from the usual watch-dogging we do here at the Sunlight Foundation. My team's goal was to take our combined skills of web development, research and story telling and create a product very different from the usual web applications and data tools we usually strive for.

I was lucky enough to be on a team with Daniel Cloud, Ethan Phelps-Goodman and Eric Mill. Originally, the four of us struggled to come up with a project that would be topical, technical and entertaining. After an extended brainstorming session where we considered projects surrounding campaign finance, the London riots and natural disasters around the world, we decided to create the ultimate data visualization using (drum roll, please) Jell-O! To be clear, our idea was not inspired by the London artists that sculpt things out of Jell-O. Our use of the jiggly substance was completely coincidental.

As we talked about what we could build and what would be of interest, we kept in mind that this year’s competition, unlike last year’s, was not limited to building applications. Our end result could be, and was encouraged to be, tangible. So when we considered mapping areas recently hit by earthquakes -- D.C., Denver and California (of course) -- it occurred to us that we should not only map those areas, but also make those maps dynamic by making them light up, and vibrate, too!. We investigated ways we could embed LED lights in a three dimensional Jell-O mold, and quickly ran into several obstacles. It looked like the Peggy 2 was going to be our LED board until we realized we’d have to solder 625 LEDs. Given the time and skill level that would require, it didn’t seem realistic. We then gave up on embedding LEDs in Jell-O and decided to go for the more obvious choice, a layer of Jell-O in the shape of the United States on top of a horizontally-oriented LCD monitor.

So it was set: we were going to use sophisticated mapping software new to all of us (especially me, since I’m not a developer) to map earthquake and other government data and then distract our audience entirely by putting a sticky mass of gelatin on the table and somehow, someway, make it jiggle on cue.

To create the underlying map visualizations we used the TileMill mapping stack from Development Seed. We collected dentist and diabetes data from the Centers for Disease Control (CDC) to map the change in obesity rates over time and the number of dentists per capita. We mapped earthquake data using information from the U.S. Geological Survey (USGS).

TileMill lived up to its promise of providing an easy to use complete solution for people with little experience mapping, as was true for most of us. Once the maps were designed we exported them to static images and displayed them as a slideshow.

Once the slideshow was created we chose a large monitor to display it on and wrapped the whole thing in Saran wrap. After we sculpted the states using Jello, we placed it on the protected monitor and displayed the maps we generated beneath the translucent dessert. When it came time to display the earthquake data, we had to resort to an over-sized neck massager to get the Jello to jiggle. Our early experiments involved installing vibration motors from a Play Station 3 controller into a layer of Jell-O. While it was definitely a sight to behold, we didn’t get the range of motion we had hoped for. Ultimately, making it shake was not an easy task.

The final product looked very much like an early prototype for a more sophisticated device, the one we had imagined in our planning phases. Perhaps intrepid tinkerers will take what we learned and build upon it to form something bigger, better and more delicious.

I’ll spare you all the suspense and let you know our team, the J-team, didn’t win. Keep checking this blog for a post by the winning team. Everyone who works in a office will appreciate their creativity and problem solving ability.

Continue reading

For this year's Labs Olympics I was on an all-star team comprised of Aaron, Alison, Tim, and myself, better known as the Labs Olympics Winners (note: we did not win, this was just our team name). Alison has a young baby at home and Aaron was out during our first brainstorming session for the birth of his niece so it wasn't a big surprise that we wound up with a plan to make a sophisticated baby monitor. (It might come as even less of a surprise that we named it How Is Babby in honor of an infamous web meme.)

At first all we knew is that we wanted to use some random gadget or assortment of Arduino sensors to give geek parents a way to monitor their geek children, but it wasn't until we realized we had a spare Microsoft Kinect sitting around the office that we realized exactly how far we could take it.

Kinect

The Kinect is an impressive device, sporting 4 microphones, RGB and IR cameras, an additional depth sensor, and a motor that allows vertical panning. Getting the Kinect running on Linux is a fairly well documented process. We leaned heavily on instructions from the OpenKinect community, which worked pretty much without issue. After doing the usual

cmake, make, make install

dance, things worked without issue on Ubuntu 11.04.

Also included in the OpenKinect source tarball are bindings for a half dozen languages, including Python. Having a Python wrapper made things incredibly easy to experiment with as I had access to python OpenCV bindings for displaying image data and NumPy for manipulating the matrices that the Kinect driver returns.

With these tools in hand we just had to decide what we actually wanted to get from the Kinect. We decided to take regular snapshots to present via a web interface, and also have a mechanism for the Kinect process to notify the web client when there was motion. Snapshots were extremely easy: with just a single line of code, we were able to bring back the RGB image from the Kinect's main camera and convert it to a suitable format using OpenCV. Once we made the discovery that there was also the option to bring in an IR image, we added a night-vision mode to our application as well. This way, the parent can adjust the camera to either take a standard image in normal light situations or switch to the IR camera for the night. (Due to a hardware limitation of the Kinect, it is impossible to use the RGB and IR camera at the same time.)

Given the uncertainty in the amount of available light and the fact that the depth sensor provided simpler data to work with (essentially a 2D matrix of depth values refreshed about 30 times per second), we decided to use the depth sensor to detect motion. NumPy's matrix operations made this a breeze. By averaging the depth of the frame and comparing the deviation across a range of frames, we could flag each individual frame as likely containing motion or not. Depending on the desired sensitivity of the alerts, the application would wait for anywhere from ten to thirty frames of consecutive motion before notifying the web application that the baby was on the move.

The Web Application

As opposed to a traditional baby monitor, which has a dedicated viewing apparatus, we liked the idea of a web console that could be viewed from anywhere, including via a mobile device. The main features of the web app would be viewing, motion alerts, and configuration of features such as SMS notifications and nightvision. The basic web app was built with Django, but we used a few add-on libraries to help accomplish our goals in the two days given for the contest.

We decided that the easiest way to get images to the user was to have the web page embed a single image that the monitoring software would update at a set interval. We used Socket.IO for a very light-weight solution to keep the image updated to the latest version. In the best case scenario, i.e. the user's browser supports it, Socket.IO will use WebSockets to keep the connection open, but will degrade gracefully and fall back to AJAX or other means to get the job done.

Because our team lacked a designer, we used a CSS framework to take care of cross-browser issues and provide some pre-designed UI elements. Twitter just recently released their Bootstrap framework, so we went with it. It styled all of the UI elements on our site, including a navigation bar, alert boxes, buttons, and a form. Although we had some unresolved trouble with the form elements not lining up properly with their labels, it proved very easy to work with, overall.

The remaining technical component of the website was the AJAX alerts on motion events detected (and logged in a DB table) by the backend. There were a few criteria for how it needed to work, the most important being that alerts needed to be somewhat persistent to the user, so that a user couldn't miss an all-important alert saying that the baby was moving, just because they were clicking quickly between pages on the site, for instance. This meant that we needed something more sophisticated than Django's inbuilt messaging framework (django.contrib.messages). The answer came in the form of django-persistent-messages. It was built to work right on top of Django's messaging system, so it worked seamlessly and was a no-brainer to set up. With django-persistent-messages working, alerts now would not disappear unless dismissed by the user, hopefully averting any potential baby-on-the-move mishaps.

In the end, there were a few features we had to leave unfinished to get the project out the door on time, including audio monitoring and SMS messaging, but we were pretty happy with the results. As usual, all of our code is available on GitHub: How Is Babby.

Continue reading