Many federal agencies still have a nasty habit of giving up control of valuable data and vital technology to contractors instead of sharing it openly with the public that pays for it.Continue reading
Recently, I gave a workshop at Open Data Day whose goal was to demystify JSON. I searched, but couldn't find any tools to do this that worked inside the browser. The only solution was to make a new one!Continue reading
Disclaimer: The opinions expressed by the guest blogger and those providing comments are theirs alone and do not reflect the opinions of the Sunlight Foundation or any employee thereof. Sunlight Foundation is not responsible for the accuracy of any of the information within the guest blog.
Derek Eder is a co-founder of Open City, a group of volunteers in Chicago that create apps with open data to improve citizen understanding of our government through transparency, and owner of DataMade, a civic technology and open data consultancy.
City councils shape nearly every aspect of city life, from what kind of canopy you can have on a storefront, to how much we pay in taxes, to the number of cops on the street.
Unfortunately, it is hard for citizens to keep tabs on what their city council is doing. A few years ago, if you wanted to be informed about a city council’s actions, you had to go to the clerk’s office and page through the hundreds or thousands of bills that were added or updated every month.
In recent years, many city clerks have taken a big step forward by publishing this legislation online. However, the current generation of municipal legislative information systems are mainly built to help councilmembers and clerks’ offices manage legislation. They were not built to help the public to understand what their city council is doing.
Well, like so many of our problems, now there’s an app for that: Councilmatic.
On Thursday June 6th at the Personal Democracy Forum (an annual conference exploring technology’s influence on politics and government), New York City’s Comptroller John Liu announced that the code behind Checkbook NYC 2.0, the city's transparency spending web portal, had been open-sourced and made available for forking on Checkbook NYC 2.0's github page. This is significant because (1) Checkbook 2.0 is enormous: it makes over $70 billion dollars in New York City spending available online in a timely, structured, and human-readable form, demonstrating that best practices in data disclosure can be followed even at scale; (2) it marks a shift to proactive civic application-sharing, by the way of the municipality’s desire to share the resources they’ve developed with other local (and even state) governments and NYC’s partnership with common municipal software vendors in this endeavor; and (3) it raises questions about what’s next for government transparency tools, civic software partnerships, and reuse.Continue reading
Disclaimer: The opinions expressed by the guest blogger and those providing comments are theirs alone and do not reflect the opinions of the Sunlight Foundation or any employee thereof. Sunlight Foundation is not responsible for the accuracy of any of the information within the guest blog.Jason Hibbets is the project manager at Red Hat and lead administrator for opensource.com. He has been applying open source principles in neighborhood organizations in Raleigh, NC for several years, highlighting the importance of transparency, collaboration, and community building. Follow the rest of his thoughts at @jhibbets. My latest writing project has been quite challenging. At the beginning of 2013, I wrapped up the first draft of a book I’m writing about the open government movement in Raleigh, North Carolina. The City of Raleigh has made a lot of progress over the last two years, which is part of the inspiration for collecting Raleigh’s story. The movement towards a more open and transparent government started to accelerate after the city council unanimously passed an open government policy. Raleigh is on the verge of defining their open data policy and a draft of their open data standards is currently posted on Open Raleigh. From my conversations with Jason Hare, the Open Data Program Manager for the City of Raleigh, the city is about to strategically release a bunch of open data. All this is in preparation for an upcoming Triangle Datapalooza, a region-wide event rumored for later this spring that aims to excite the entrepreneurial community about open data and discover new opportunities. This is all very exciting for civic geeks and hackers in the Triangle area. I’m excited because I saw an opportunity to collect Raleigh’s open government and open data story. I’m in the final stages of finishing the book. The first round of editing is complete and my editors and I are finalizing the latest changes. I plan to self-publish the book (paperback and eBook), and I’m considering starting an IndieGoGo campaign to help crowdfund the initial round of publishing. I am also crowdsourcing ideas for the book cover on my personal blog. Continue reading
Disclaimer: The opinions expressed by the guest blogger and those providing comments are theirs alone and do not reflect the opinions of the Sunlight Foundation or any employee thereof. Sunlight Foundation is not responsible for the accuracy of any of the information within the guest blog. Jennnifer Wike is an Editor and contributor for Opensource.com, a community service website of Red Hat dedicated to highlighting the ways in which the 'open source movement' is shaping government, law, education, science and technology, and other areas of life. Jen also helps other businesses develop their content strategies and blogs about growth in downtown Raleigh, NC where she lives. Follow her on Twitter or you can contact her at firstname.lastname@example.org. The open government movement in our country is well underway, though still brand new in terms relative to the pace of the workings of government. Change tends to be delivered slowly, as evident during President Obama’s re-election campaign this year when many of us had to remind ourselves that though some change has trickled down over the past four years, much of it has yet to come to pass due to the inherent processes of government bodies. And yet, it still astonishes me how quickly ‘open’ ideas are being accepted, built and implemented into city governments from the east to west coast.Continue reading
David Eaves has a thoughtful post over at TechPresident talking about open source and the transparency community's commitment to it -- a commitment that David sees as half-hearted. Sunlight's mentioned in the post, and the MySociety initiative that prompted the post is something that our team has been thinking about a lot. I think there's something to David's criticisms. But he's missing a few important things.
But let's get the baseline stuff out of the way first. Sunlight loves open source. Our whole stack is built on it, from the Varnish cache your browser connects to, to the Django/Rails/Flask/Sinatra/whatever app behind it, to the Postgres/Mongo/Redis/Solr/elasticsearch datastores that power it, to the OpenOffice suite that edits the grant application that paid for it all. All of our code is up on GitHub, and we welcome and celebrate contributions from the community.
But, Kindle contest aside, the above examples are mostly about us benefiting from open source. What have we done for the movement lately? This is the crux of David's critique:
So far, it appears that the spirit of re-use among the big players, like MySociety and the Sunlight Foundation, only goes so deep. Indeed often it seems they are limited to believing others should re-use their code. There are few examples where the bigger players dedicate resources to support other people's components. Again, it is fine if this is all about creating competing platforms and competing to get players in smaller jurisdictions who cannot finance creating whole websites on their own to adopt it. But if this is about reducing duplication then I'll expect to see some of the big players throw resources behind components they see built elsewhere. So far it isn't clear to me that we are truly moving to a world of "small pieces loosely joined" instead of a world of "our pieces, loosely joined."
I think David's missing a few important examples. For one thing, Sunlight's been adopting and investing in other organizations' code for a while now. PPF's OpenCongress has long been a Sunlight grantee, of course, and their code is entirely open source, including specific components like Formageddon that we commissioned. It's been more than a year since we began providing support for the Media Standards Trust to open-source and continue to develop SuperFastMatch; that's a partnership we think has tremendous potential to benefit both us and others, and you can expect to see some additional collaborations announced soon. Politwoops is a recent example of Sunlight adopting, extending and then launching a project started by another NGO -- the Open State Foundation, in this case (we're in the process of working with them to open-source the code).
But this is at the level of fairly specific partnerships with other transparency NGOs. The fact is that the more specific a project's use case, the harder it is to generalize its adoption. The more fundamental and abstract a tool is, the easier it is to adopt it and contribute back to it. It's no coincidence that we have people on our team who have patches in the Linux kernel but none who have patches in FixMyStreet. We see plenty of people use our Django apps and middlewares, but (so far) no successful redeployments of Influence Explorer. We've contributed a number of patches to the Boundary Service project that David mentions, but none to Ushahidi. Heck, back in my fixed-width font days, even I managed to get a minor patch into PySolr.
It simply gets harder to collaborate when you move to a less-abstract level of software. Requirements become more specific, and there cease to be good, general approaches to tackling problems. I saw this first-hand when I threw together the Elena's Inbox project. That effort generated a lot of excitement from other folks who had access to email archives, and we were glad to speak to all of them. I was eager to offer advice, answer questions and generally do some hand-holding, but I found myself wishing I had better news for the people who got in touch with me. Because unfortunately the reusable part of the site isn't all that valuable -- it's just some ugly templates and a basic Django app that provides endpoints for search and starring of emails (though we do have some much less ugly templates waiting for the next time we do a similar project). The real work and value-creation comes in the weekend following the government's Friday afternoon email document dump, when you need a programmer to lose sleep writing endless regular expressions that parse the idiosyncratic formatting of what's likely to be a badly-OCRed pile of text, then apply algorithmic approaches -- usually specific to the particular document set -- to stitch individual emails back together into threads. Come Monday morning, you'll be facing a huge, all-hands-on-deck manual review process as your staff tries to collapse duplicate entities down to single individuals (a process that can be aided by some string-similarity techniques, but which inevitably involves a lot of judgment calls and contextual knowledge).
Setting up an EI-style-site is unfortunately never going to be a clean, easily-repeatable process; not until government starts releasing MDBs or exposing IMAP endpoints (something we have yet to see, as far as I know). And this is fairly typical of work in our space: a lot of it needs to be purpose-built because of the quirks of government and the datasets it produces.
The good news is that although our movement is still quite young, we've already learned some lessons. I think MySociety's components strategy reflects this: they're moving down a layer of abstraction -- cautiously and after much consideration -- and tackling a slightly-more-specific task than a typical NOSQL or GIS project; a task that's still abstract enough to be reusable, but which is targeted enough to be particularly relevant to transparency organizations. It's something that we think is worth pursuing, and that we're anxious to help to make into a success. It probably won't make sense to spend time replacing Sunlight's too-specific-to-be-reusable but perfectly-useful-for-us entity store with PopIt in the near term. But those organizations that come to this space after us should be able to benefit from the lessons learned by MySociety, Sunlight and others. It's the same reason why Open States has been refactored twice: it takes time and experience to figure out what parts of a problem can be abstracted and made reusable.
There's no question that we can do better. We're looking at which projects have the most potential for reuse, and -- where appropriate -- we're planning to clean up their docs, add easy Heroku deployment support, roll some AMIs, and support some up-and-coming general source data formats. We'll also be taking a hard look at how our APIs are organized: we can make our data more easily reusable, too.
But specificity is often the enemy of reusability, and we think some of the most interesting opportunities tend to involve very specific problems. It's a real tension, but one that we're committed to continuing to work to address.
UPDATE: MySociety's Tom Steinburg has also posted a response to David, in which he explains the rationale behind MySociety's components strategy in considerably more detailContinue reading
Today, our guest post is written by Joshua Gay, a programmer, activist, and community organizer whose interests revolve around technology,... View ArticleContinue reading
Since I started the Tools for Transparency post back in July, I’ve written about quite a few social media resources... View ArticleContinue reading