Today we’re launching Clearspending — a site devoted to our analysis of the data behind USASpending.gov. Ellen’s already written about this project over on the main foundation blog, and you should certainly check out her post. But I wanted to talk about it a little bit here, too, because this project is near & dear to my heart, having grown out of work that Kaitlin, Kevin and I did together before I stepped into the role of Labs Director.

The three of us had been working with the USASpending database for a while, and in the course of that work we began to realize some discouraging things. The data clearly had some problems. We did some research and wrote some tests to quantify those problems — that effort turned into Clearspending. The results were unequivocal: the data was bad — really bad. Unusably bad, in fact. As things currently stand, USASpending.gov really can’t be relied upon.

You can read all about it over at the Clearspending site, and I hope you will — in addition to an analysis that looked at millions of rows of data and found over a trillion dollars’ worth of messed-up spending reports, we spent a lot of time talking to officials at all levels of the reporting chain. I don’t think you’re likely to find a better discussion of these systems and their problems.

And make no mistake, these systems are important. If you want to find out — in detail — what the government is spending your money on, these are the systems you’re going to have to turn to. In fact, when Congress wants to know how it’s been spending money, they typically ask CRS to have a look at what’s in these systems. There are other measures of spending — the budget, for one, and Treasury data about expenditures for another — but that data tends to be less detailed or impossible to organize by program. This isn’t just about one website — it’s about the way we keep tabs on our society’s use of its resources. We shouldn’t let these systems become afterthoughts. Regardless of whether you think the government spends too much of our money or too little, surely we can all agree that taxpayers deserve to know how much it’s spending right now.

But there’s also a larger point that I want to make about this analysis: it’s not news — not really. People have known that this data is bad for a very long time. Not in this level of detail, perhaps. But they’ve known there were problems. Yet instead of fixing those problems, the folks behind USASpending decided to build new websites on top of the same lousy data, collecting superficial plaudits from advocates who were excited about the apparent potential of the site but didn’t have sufficiently direct access to the data to see its flaws

From one perspective, this is understandable. These are hard problems, and the people responsible for USASpending don’t really have the power necessary to get other agencies into line. Still, it’s deeply disappointing to me that this could happen. And it’s a reminder that we shouldn’t get complacent — that it’s great to hear government officials say the right things about transparency, but that we still need to pay careful attention to the results of their efforts. This is one of the reasons I think it’s vitally important for those of us advocating for open data need to be open data users, too. If we don’t actually get our hands dirty with this stuff, we as a community are not going to be able to offer meaningful critiques.

Finally, I want to acknowledge the help we’ve received as we pursued this project. Donna Fossum may know more about these systems than anyone else on the planet; the incredible depth of her knowledge benefited the project immensely. We’re also grateful to OMB Watch for contributing their perspective, and for being the first ones to let us know that USASpending had problems worth investigating. And many thanks to the officials at GSA, OMB and HHS who were generous with their time and anxious to hear about our work.