We're excited to announce Upwardly Mobile, Sunlight's new webapp funded by the Knight Foundation that allows you to research where in the country you could enjoy financial security and an improved quality of life. Upwardly Mobile is an easy-to-use relocation research tool backed by powerful economic data, allowing granular comparisons without digging through arcane government reports for each indicator. We sifted through all this data so you don't have to, and this information is now presented seamlessly on any mobile or tablet platform.

Just enter your zipcode, career information and cost-of-living importance and then Upwardly Mobile gets to work generating a list of ideal places for you to move. Alternatively, you can browse individual cities to compare them to national averages. Through charts and graphs, you can explore how metropolitan areas of similar size compare to where you live now, including:

• Occupation: Both the average salary for the selected occupation over time and income data for the entire metropolitan area.

• Housing costs: Rents, as well as maintenance services and goods such as furniture and appliances.

• Cost of living: Apparel, education, food and childcare.

• Quality of life costs: Recreation, transportation and health care.

Part of putting this responsively designed app together included deciding which economic factors make the greatest difference in people’s lives. For instance, we decided that salary and housing costs are more important than other economic indicators such as the cost of recreation services. These weights impact the base ranking, but the importance attached to each economic category can be changed by your selections in the survey. For more information on this methodology and the technical background, check out my colleague Jeremy's blog post here.

The Upwardly Mobile app utilizes data comes from many sources: Bureau of Economic Analysis, the Federal Financial Institutions Examinations Council, Bureau of Labor Statistics, the National Association of Child Care Resource and Referral Agencies and the U.S. Census.

Upwardly Mobile is the second in a series of National Data Apps, developed with support from the John S. and James L. Knight Foundation. The first was Sunlight Health, which helps people make more informed decisions about medical care. Sunlight also created mobile apps for monitoring lawmakers: Congress for Android and Windows phones and Real Time Congress for iPhone. The recently launched OpenStates app for iPhone and iPad tracks the inner-workings of all 50 state legislatures.

Note: 4/3/12, 5:25 p.m. This post has been updated to clarify the weighting of salary and housing costs against other economic indicators.

Today Darrell Issa, chairman of the House Committee on Oversight and Government Reform, spoke about his vision for improving open government and federal spending transparency at the O'Reilly Media 2011 Strata Summit. The Chairman focused his remarks on the DATA Act, the bipartisan legislation he introduced that would transform how government tracks federal spending and identifies waste, fraud, and abuse.

He emphasized the importance of making government data available online in real time so that innovative minds can immediately make use the information to build their own businesses. Business, in turn, would help the government identify program mismanagement and data quality problems. The Chairman specifically singled out Vice President Biden as a supporter of efforts to find a common solution to make data available in a systematic way.

In an ensuring Q&A with O'Reily Media's Alex Howard, Chairman Issa explained that the private sector must step up as advocates for greater openness because they will benefit from building and using the tools made possible by greater transparency. He added when government drives down the cost of obtaining information, private individuals will derive value from the analysis of data, not its ownership.

The Chairman also addressed proposed cuts to the Electronic Government Fund, which supports many government transparency programs, saying that he doesn't always agree with funding cuts. He added that he has an assurance from appropriators that they're willing to listen when he makes that case that increased spending on transparency programs in the short term may save more money later. Accurate data will help drive down the inefficiencies of government.

Yesterday the government’s Federal Awardee Performance and Integrity Information System (FAPIIS) came online. This is something we’ve been looking forward to for a while. It’s easy to find horror stories about the mismanagement of contracts; this isn’t surprising when you consider the disorganized constellation of contractor oversight databases that exists, many of which aren’t open to the public. Getting FAPIIS online should be a step toward fixing that problem. Yesterday government took that step.

POGO has some thoughts about it that are certainly worth your time. But we can’t help chiming in as well. In short: this site is terrible. As one colleague said, “This might be the worst website I’ve ever seen.”

This is at least debatable. Contracting databases are part of the world of procurement, procurement is heavily influenced by the Defense Department, and DoD has a proud heritage of producing websites so ugly that they make you want to claw out your eyes. So FAPIIS has company. But if this was just a question of aesthetics, we wouldn’t be complaining.

Assuming you’re using one of the few web browsers in which the site works at all (Chrome and Safari users are out of luck), the experience is off-putting from the start, as users are warned that their use of the site may be monitored, surveilled, or otherwise spied upon (you don’t necessarily surrender your right to speak privately to your priest by using the website, though—thanks for clearing that up, guys!). Perhaps this is why their (arguably superfluous) SSL certificate is utterly broken. But let’s click past the security warnings and proceed.

Here’s the next screen. It contains a captcha.

Let’s be clear: the use of a captcha to gate government data is outrageous. Government should be making its data more accessible and more machine-readable. Captchas are designed to interfere with automated tools that facilitate malicious acts. But downloading government data is decidedly not a malicious act. Why are we trying to limit machines’ ability to use this data?

But our irritation with the captcha is softened a bit by how laughably inept its implementation is. It’s made of black and white text, unrotated, unskewed, superimposed on the same black and white grid every time. Here’s a stab at how you’d beat it:

1. Subtract grid
2. Flip every white pixel that’s bordered by 2 or more black pixels to black
3. Identify columns of all-white pixels and slice the image by them
4. Crop the resulting slices, then recombine
5. OCR

You could probably get this done using a stock PHP distribution in about an afternoon. But you don’t need to, because even this pathetic level of security isn’t properly implemented! Instead the human-readable text is sent to the client as a SHA1 hash in a hidden field. That hash is compared to the hash of what the user enters for the captcha code. So a scraper can just ignore the captcha and resend a solved hash for every request — it’ll work just fine¹. They didn’t even salt the hash. Whoever wrote this has absolutely no idea how to implement a secure system.

After the captcha, things start to get really weird, with radio buttons with onclick handlers being used as hyperlinks. It’s unclear to me whether the programmers responsible for this interface had ever actually used the web or simply had it described to them. Either way, whoever built this should be embarrassed. Whoever managed the project should be embarrassed. Whoever signed off on delivery should be embarrassed! But we haven’t even gotten to the worst part yet.

That’s because while all of the above will be embarrassing to any developer who takes pride in his or her craft, the quality of a government website is ultimately less important than the data it exposes. And there is no FAPIIS data in FAPIIS. Not yet, anyway. Such data exists, mind you. But the decision was made not to include any historical data when FAPIIS went public. Presumably the contractors who did a bad job, and who were reported for doing so, are concerned that people might look at those reports and get the impression that, uh, they did a bad job. Others may be concerned that the database could cast them in a bad light and raise uncomfortable questions. That government caved in to the demands of these vendors — vendors who are supposed to be serving government! — can only be described as an act of craven capitulation. We’ve FOIAed for this data, and if we’re lucky, perhaps we’ll even get it. But it ought to be online right now.

As a matter of principle, it’s good to see government opening closed databases, and Congress deserves credit for deciding to open this one. But what has followed that decision deserves only whatever the smallest quantity of plaudits is that’s still distinguishable from zero. I hope that the site removes the captcha, offers bulk downloads, and fills up with useful, unsanitized data. But whoever built this travesty deserves to have an entry in FAPIIS of their own.

1: You do need to update the JSESSIONID cookie and get a fresh value for the org.apache.struts.taglib.html.TOKEN hidden variable, but this is easy enough to do.

Cross-posted from the Sunlight Labs blog

As I noted in my speech yesterday at the Gov2.0 Summit, Gov2.0 has become a popular catchphrase in Washington today and no organization has been more excited about its potential when it comes to data transparency than the Sunlight Foundation. But now, some 20 months since President Obama made his initial commitment to technology and transparency, we have numerous concerns. One of the core examples that I used was USASpending.gov, which President Obama championed into law when he was in the Senate, along with Sen. Tom Coburn, in 2007.

USASpending was created to provide the public with information about how the federal government spends our tax dollars. It was launched nearly three years ago and has already gone through three redesigns, each one flashier than the next. The site is pretty impressive graphics-wise, but unfortunately the data provided is full of inaccuracies, according to Clearspending, a new Sunlight site that tracks and illustrates just how broken the data is. We were deliberate in our approach when conducting this analysis, and we hope that by giving these problems the light of day, it might actually help get them fixed.

Think of Clearspending.com as a kind of scorecard that analyzes how well U.S. government agencies are reporting their spending data on USASpending.gov.

What Sunlight has found, and Clearspending shows in great detail, is that just under* $1.3 trillion in federal reporting data from 2009 is unreliable. The data inaccuracies we uncovered account for 70 percent of the total $1.9 trillion in government spending data reported in that year. Some of the numbers are too big, some are too small and some are missing completely, while other spending data entries don’t have the detail that’s required or were reported months later than the law demands.

When it comes to making data available, it has to be accurate. Federal agencies need to focus first on the quality of data they collect. If the data is unreliable, then the quality of websites they release — or the tools built upon it  — is irrelevant.

The government has known about the problems we’ve illustrated on Clearspending, and they say they’re working to fix it. But instead we have only gotten a series of redesigned websites, each one with data just as unreliable as the one before it.

There’s a tremendous amount of work left to do before Gov2.0 becomes a reality. These are not easy tasks, and certainly not glamorous ones. But these are the types of challenges that we must undertake if the promise of Gov2.0 is going to be realized.

*We updated this number - at launch it was just north of $1.3 trillion. You can see why here.