About a year ago, Sunlight, EFF and over 150 civic hackers reverse-engineered Congress’ contact forms to create Contact-Congress – an open source project under the @unitedstates umbrella on GitHub. This project, as previously highlighted, aims to codify (in an open format) the steps necessary for submitting electronic messages to any member of Congress, opening the door for the development of diverse tools and applications for facilitating dialog with representatives. Indeed, if you’ve emailed your representatives on OpenCongress, then you’ve already benefited from the project. Its inception and continued upkeep play an important role in minimizing barriers to constituent communication while promoting a more accessible and open government.

Over the past year, we’ve received feedback for our implementation of Email Congress and have been working to identify the issues and concerns of our users. In the coming weeks we will be deploying a revamp of the project that will significantly improve the entire experience of sending emails to members of Congress. This post will detail some of the technical highlights of our implementation, so others may learn more about our development process and what it takes to deploy an email service for Congress. For reference, you can view the open source code here.

Overview

As the GitHub description reads, the project is a “Lightweight Flask web app to courier email messages to phantom-of-the-capitol using postmark.” It works by stitching together a number of APIs and user input for the ultimate objective of transforming an email message into an automated submission of a congressional web form. There are a number of steps and challenges along this road, so perhaps tracing the course of a typical user email would best illustrate how each one is overcome.

Step one: User sends an email

The user sends an email to a particular member of Congress at an @opencongress address, such as Sen.Reid@opencongress.org or Rep.Boehner@opencongress.org. Sunlight’s email accounts are managed by Google, so we set up a filter to forward messages to our private inbound postmark address as shown below.

The postmark account is then configured with an inbound webhook to make a post request to our web app each time an inbound event occurs (when Gmail forwards the email). The URL we provide here is simply the endpoint /email/postmark/inbound for our app, which is set up to process the postmark JSON data containing the email’s subject, body and metadata. This request is secured by basic access authentication to prevent unauthorized data from reaching our app.

Step two: Reaching our app

If all goes well, then the email data should reach our app. At this point, if you’re not familiar with postmark, then you can skim over the relevant Email API documentation to see an example of how the email data is structured as JSON. We first make use of the postmark-inbound-python project to convert the JSON into a Python object, which allows us to interact with field data more conveniently. Next, a number of things may occur depending on whether the user has previously registered, is exceeding rate limits or is a first-time user. For completeness, let’s continue with the pathway of first-time users.

We can’t immediately submit messages to congress from first-time users because congressional web forms require address and contact information to verify that the message is indeed coming from a constituent. Thus, first-time users will receive a reply to their original email, welcoming them and asking them to click a link to complete a registration process where we gather this information. We make use of the python-postmark project to handle routing emails through postmark. In order to thread the emails properly, we make use of the email headers “In-Reply-To” and “References” as well as the traditional subject “Re:” prefix as email clients can vary in level of scrutiny necessary for threading. You can see a code example for how we accomplish constructing such a postmark email below.

    return PMMail(api_key=settings.POSTMARK_API_KEY,
                  sender=cls.SENDER_EMAIL,
                  to=user.email,
                  subject='Re: ' + msg.subject,
                  html_body=render_without_request("emails/validate_user.html",
                       context={'verification_link': veri_link,
                                'user': user}
                 ),
                 track_opens=True,
                 custom_headers={
                     'In-Reply-To': msg.email_uid,
                     'References': msg.email_uid,
                }
                )

Step three: Collect data and determine district

The user clicks the link in the reply email and is taken to a page where we collect necessary contact and address information, including prefix title, first name, last name, address, city, state, zip code and phone number. With this information, we are able to use Sunlight’s Congress API to determine the user’s congressional district. If a zip code overlaps multiple districts, then we make use of Texas A&M Geoservices to obtain a latitude/longitude via geolocation, which we can then pass to the Congress API to definitively determine a congressional district. The accuracy of this step is imperative as congressional web forms will ultimately reject messages if the provided address information indicates the sender resides outside the corresponding district.

Once we have a state and congressional district, then we can match the user with their senators and representative. Sometimes this won’t match with the original intended recipient as congressional district boundaries change fairly frequently or the user may be trying to email a member of congress that doesn’t represent them. We thus provide as many explainers as possible to help the user understand how we’ve matched them and why they can only message certain representatives. The user can then decide if they wish to continue by selecting which representatives they wish to send their message and clicking a send button.

Step four: Automate form submission

If the user decides to send their message, then a job is passed to Celery using Redis as the broker. The job is responsible for making an API call to a phantom-of-the-capitol instance with the email contents, address and contact information, and desired recipients. The phantom-of-the-capitol instance handles submitting the data via simulating user input with a headless browser such as PhantomJS. This is a huge upgrade from our original form-submission implementation, formageddon, because it allows navigation of web forms that require JavaScript, whereas formageddon was built on top of Mechanize and couldn’t process JavaScript. This should allow for higher success rates in form submissions as congressional websites have modernized considerably and many are starting to require JavaScript to submit their forms.

In some cases, a message will be unable to submit successfully due to unexpected technical complications, such as website maintenance or server downtime. In these cases, the user will be notified by email of the situation, and we will provide links to forward them to the contact form of the intended recipient. Otherwise, the user will receive an email notification that their message was successfully sent.

Step five: Freely message members of congress

Now the user is signed up and can freely send emails to their members of Congress without going through the signup process again. If a user moves and wishes to change their address information, then there is a link at the bottom of every email notification that will take them to a page that will allow them to do this. This makes it so the entire registration and authentication system is a lightweight email-based process using unique URL tokens; no need to keep track of yet another username and password!

Final remarks

Hopefully this post sheds light on how our new system of emailing congress will work overall and all the moving parts propelling each step. As the reader should see, transforming an email message into a congressional web form submission isn’t a trivial task and relies on a number of APIs and open source projects along the way. Sunlight is truly appreciative for all the great work and people who provide these resources.

Perhaps some day Congress will release a more convenient means of contacting representatives, but until then, keep your eyes peeled for the launch of our revamped Email Congress project in the coming weeks!