Yesterday, we told you (and the New York Times published a longer article) that USDA published the Social Security numbers of individuals who receive federal aid in a publicly available online database of government grants. That information was inadvertantly pick up in a database that we funded — FedSpending.org. The database, which details government grants and contracts in a user friendly way was developed by OMB Watch. It's a hugely popular database — during the month of March, there were roughly 1 million searches made. (That's not visitors or hits, that's 1 million people looking up stuff.)

This morning, Gary Bass, OMB Watch's Director, sent a note to me and several other funders, to tell his side of the story explaining how it all unfolded. It's worth a read:

Once we knew the New York Times was going to break the story on its website yesterday, we prepared a detailed description of events that transpired. That information, along with a statement from OMB Watch, is available on FedSpending.org as well as our home page….

In summary, when the user notified us that her Social Security number was available from our site, we verified that the data was the same as on the data we obtained from the Census Bureau and suggested she contact officials at Census and USDA, the agency where she received a loan. That evening we redacted the data field from her record and suggested to government officials that this may be part of a larger problem.

Late on Monday, we received a request from the government to redact the data field from the entire database for 30 days while the government notifies other agencies. (The Census Bureau noted that it was redacting the field from the files that can be downloaded from its site. On Tuesday, after the government's redaction, we found the same data with the Social Security numbers on a National Archives website, highlighting the extent of the problem for government. That information, as of this morning has now been restricted — meaning the web site doesn't display the SSN anymore.)

We responded on Monday that we would redact the data field for 30 days if the government provides a plan (within the 30 day period) for correcting the data for the field in question . The data field, called the Federal Award ID, is a unique identifier for specific financial transactions. (What two agencies within USDA were doing was including the 9-digit Social Security numbers as part of the 15-digit ID. It does not appear any other agency in government did this.) Without the number, it is near impossible to track specific grants, loans and other forms of financial assistance. For example, you need that identifier when doing a Freedom of Information request for specific financial transactions.

In other words, we were not prepared to let government's apparent violation of federal law (by releasing Social Security numbers) become an excuse for reducing transparency and accountability about government spending. Plus we knew that it is not a hard technological fix (the government could generate new numbers and give us a crosswalk to post the corrected numbers). (Of course, this doesn't fix the problem for those who downloaded the files from government websites for around the last 10 years.)

Although there was lots of conversation with government on Fri (4/13) and Mon (4/16), after we sent our letter, there was no more discussion with us. (Well, there was one. The USDA Chief Information Officer sent me an email, apparently accidentally, telling the Commerce Department to have legal counsel tell OMB Watch that it is violating the Privacy Act. BTW, the Privacy Act only applies to government agencies such as the USDA.)

Accordingly, we talked confidentially with colleagues in the news media about whether we should voluntarily redact the data in order to asses the damage. In doing so, two felt this was an important story to tell; one was from the New York Times and the other from the Sunlight Foundation, the organization that provided funding for FedSpending.org.

By Wednesday night we told the two organizations we would not stop them from doing a story, but asked if they would wait until Friday (4/20) to see if government did something by then. Coincidentally, as the New York Times reporter began questioning government officials on Thurs (4/19) morning, OMB Watch received a call from OMB to discuss the issue. After a series of calls throughout the day on Thurs, by the early evening we received assurance that we would receive a written response indicating the government will provide a public plan for correcting the data field within 30 days. We received the written agreement late on Friday (4/20).

OMB Watch has temporarily redacted the Federal Award ID data field from the database. We eagerly await the government's plan to correct the problem, especially since there is now a law that requires the government to create a database like FedSpending.org by Jan. 1, 2008 (which we proudly helped push).

We couldn't be more pleased with the way OMB Watch has handled this controversy, or the wiseness of this investment in their work.