Six months old today, the Dodd-Frank Wall Street Reform and Consumer Protection Act provided vast new powers for federal regulators to collect information on financial institutions—information that could be crucial in staving off future crises, some vital elements to be released publicly. Yet the machinery of government grinds slowly, and the public has yet to see much of it.

• President Barack Obama has yet to appoint a director for the  Office of Financial Research, a new Treasury office that has been called the “CIA of financial regulators” because of its far-reaching power to gather data and information on financial institutions. The office, which is supposed to supply data and analysis for the also newly created Financial Stability Oversight Council, is also required to produce several public databases to help track financial instruments. So far the office’s main action was in November, when it published a proposed policy statement in the Federal Register on creating universal identifiers for a particular legal entity that takes part in a financial transaction.

• Bizarrely, the federal government has no way of tracking foreclosure rates. Even the General Accountability Office relied on private data sources to do  analysis of the foreclosure crisis. The new law mandates that the department of Housing and Urban Development create a new public database tracking foreclosures. However, an agency spokesman, Brian Sullivan, says the department faces legal and resource issues and that the database may be as many as three years in the making—even as the national scandal over banks employing robo-signers on loans continues to unfold.

• One of the major transparency provisions of the new law is to bring trading of derivatives—or swaps—into transparency, by requiring that such trades be reported publicly in “real-time”. The Commodities Future Trading Commission (CFTC) published proposed rules for “real-time” reporting in December. While the law requires that the final rules be published by July, the agency notes that “participants will need a reasonable amount of time in which to acquire or configure the necessary systems,” and says that some of the reporting is not expected to be made public until January 2012.

These are just a few examples from the vast world of financial regulation where vital public information is slow in coming. In the coming months, the Sunlight Foundation will be tracking closely the tender transparency points in the way our government regulates financial industries, as well as the lobbying forces that help sway the government’s actions. Stay tuned.

Just look at any one of Sunlight's projects and you'll realize that it takes a mountain of data to help keep government open and transparent.  From district information to campaign expenditures to lobbying dollars, making sense of large data sets is an intensive, concerted effort.

Many of your own projects use dozens of spreadsheets, take up thousands of rows of data and live somewhere on our laptop, accessible only to you.  This works to a point, but in an era of sharing, collaborating and web-based storage, it isn't an optimal solution.

Google Fusion Tables is an experimental project from Google Labs with the goal of making sharing and collaborating on large sets of data much simpler.  Fusion Tables isn't focused on the traditional database system that requires "complicated SQL queries and transaction processing," but is rather focused on "fusing data management and collaboration: merging multiple data sources, discussion of the data, querying, visualization, and web publishing."

Google Fusion Tables allows you handle large amounts of data: you can upload files of up to 100 MB in formats like Excel, CSV and KML. You can also programmatically update, delete, query and visualize data using their API. Plus, you can merge your own data with existing public sets, allowing you to add further value and context to your own information.

While Fusion Tables is an experimental Google project, it shows great potential in allowing the less technically savvy to easily leverage large data sets while communicating and collaborating much more effortlessly.

For more information, check out the video and related links below -

• Fusion Tables API
• Fusion Tables Examples
• Fusion Tables User Group

In many offices, when technology questions arise, the answer is to reflexively trust the technologists. These are often the folks who link to Venn diagrams of the fine distinctions between nerds, geeks, and dweebs; who prefer the comic xkcd to the Far Side; and who trust slashdot over NPR. So when it comes to the question of how the government should make information available online -- in particular, how data should be made available online -- most people's first inclination is to nod to the technologist and slowly back away. That disengagement is a mistake.

How information is made available online fundamentally controls what can be done with it. Fortunately, an intelligent layperson can understand how structure makes data usable. That's important, as the intelligent layperson is likely the one writing the rules on how government data will be made available: whether as a congressional staffer, a federal agency employee, or a citizen making a request. Awareness about data structure encourages smarter specs, and the ability to get more out of your information.

The very smart technologists at Princeton's Center for Information Technology Policy have put together short blogposts that explain key concepts, and are geared towards the intelligent layperson. Sunlight Labs is working in a similar direction. The following articles are worth considering.

• "Government Datasets That Facilitate Innovation," Freedom to Tinker Blog (3/1/2010)
• "Basic Data Format Lessons," Freedom to Tinker Blog (3/2/2010)
• "Labeling Dataset Contents," Freedom to Tinker Blog (3/3/2010)
• "Correcting Errors and Making Changes," Freedom to Tinker Blog (3/8/2010)
• "Best Practices for Government Datasets: Wrap-up," Freedom to Tinker Blog  (3/12/2010)
• "Drafting Guidelines for Government Data Catalogs," Sunlight Labs Blog (3/29/2010)

If you know of other articles along these lines, please add them in the comments.

If you've ever tried to research federal contracts you'll find that the databases used to house those contracts online are not so great. Sen. Claire McCaskill held a hearing yesterday titled, "Improving Transparency and Accessibility of Federal Contracting Databases." Nancy Scola wrote up the hearing and it isn't pretty:

All told, there are a million lines of code involved. But there's really no all told here, because the databases don't talk to one another. For example, FPDS, the Federal Procurement Data System doesn't communicate with EPLS, which stands for Excluded Parties List. Which means that theUSASpending.gov website -- heralded as the American public's window into the inner-workings of government, but powered by FPDS -- doesn't even know that contractors contained within it have been banished from government service for defrauding the United States government or otherwise behaving badly. What's more, on some of these legacy systems, a search for Contractor X, Inc. won't return results for Contractor X Inc. The shorthand for that particular wrinkle came to be known, during the hearing, as "the comma problem."

In fact, GAO's William Woods explained to the senators, the poor state of those databases meant that when his agency was asked by Congress to detail how many contractors were billing the United States government for work in Afghanistan and Iraq, the government watchdog group was forced by technology to admit its ignorance. "We could not answer those questions," said Woods. How many KBRs are at work in American war zones, being paid with taxpayer dollars? How many Blackwaters? Dunno.

The biggest problem, however, didn't turn out to be the current state of disrepair, but rather the inability to figure out what to do with the whole disclosure regime. To the surprise of almost everyone in the committee room, the General Services Administration (GSA) has been working to create a more sensible contractor disclosure regime with a more accessible public face. It was difficult for federal Chief Information Officer Vivek Kundra to identify who exactly would be overseeing the -- yes -- contract to revamp the databases. Ultimately that responsibility came down to either the GSA, the Office of Management and Budget or the Office of Federal Procurement.

As Scola writes, "Senator Robert Bennett spoke for many of us today when he sat up on the dais in room 342 of the Dirksen Senate Office Building and rubbed his temples over, and over, and over, and over again."

Public Citizen reports that a federal database will soon go online that will give car buyers information about individual cars they are considering, such as whether the cars stolen, salvaged or rebuilt. Sounds great, right? The problem is that Congress passed a law requiring the database 16 years ago.Yup. You read me right. This database was mandated 16 years ago.

Three consumer groups, Public Citizen, Consumers for Auto Reliability and Safety (CARS) and Consumer Action, had to go to court to make it happen. In September, they won a federal court decision, with the court requiring the U.S. Department of Justice to set up the National Motor Vehicle Title Information System by tomorrow (Jan. 30th), which will require states, insurance companies and junk yards to report safety and other information about individual automobiles. Some states, notably California and New York, are violating the court order by dragging their feet and not providing any information, but the consumer advocates are pressing the governors in those states to abide by the court ruling.