The White House this morning released two new documents, and one has strong implications for how important data is disclosed.

The Memo on Regulatory Compliance (not to be confused with the larger Executive Order on Regulatory Review) specifically addresses data and information about how private entities disclose information to the agencies that regulate them.  The memo describes the importance of such disclosure:

Greater disclosure of regulatory compliance information fosters fair and consistent enforcement of important regulatory obligations.  Such disclosure is a critical step in encouraging the public to hold the Government and regulated entities accountable.  Sound regulatory enforcement promotes the welfare of Americans in many ways, by increasing public safety, improving working conditions, and protecting the air we breathe and the water we drink.  Consistent regulatory enforcement also levels the playing field among regulated entities, ensuring that those that fail to comply with the law do not have an unfair advantage over their law-abiding competitors.  Greater agency disclosure of compliance and enforcement data will provide Americans with information they need to make informed decisions.  Such disclosure can lead the Government to hold itself more accountable, encouraging agencies to identify and address enforcement gaps.

To achieve those goals, the memo mandates three actions.

First, agencies have 120 days to make plans about how they're going to post compliance data better. This is an important step from the White House, whose December 2009 Open Government Directive (itself an OMB Memo) was broader, and required plans for data, without creating meaningful priorities for which types of data should be disclosed.  After some criticism, and after considering different approaches to data prioritization, this is one strategy that's emerging: to force agencies to focus on their core regulatory functions, and disclose that information better.

To do this well, agencies should come up with concrete plans. Plans to make more plans for forming non-binding working groups aren't an appropriate response.  Each agency is capable of publicly auditing all of their regulatory compliance data, to create a plan that reflects the agency's unique ability to lay out what is knowable about their work, and chart a course toward better disclosure of their core functions.

Additionally, these plans should be public, and online, on each agency's /open page.  The memo doesn't require it, but there's no reason agencies should do this planning in isolation.

Second, the CTO (Aneesh Chopra) and CIO (Vivek Kundra) are directed to work with the agencies to get their information online in "in searchable form, including on centralized platforms such as data.gov, in a manner that facilitates easy access, encourages cross-agency comparisons, and engages the public in new and creative ways of using the information."

Third, top officials are directed to "explore" better ways to share compliance information across agencies.  The vagueness of this mandate reflects the difficulty in its enactment.  The barrier to this sharing that Sunlight is most familiar with is the unique identifier problem, where entities tracked by government agencies are all assigned different unique identifiers.  The different systems government relies on to assign identity to (especially corporate) entities are largely incompatible, proprietary, and poorly designed.  Among many other problems, this needs to be addressed in order for regulatory compliance information to be more effectively shared.

Today's Memo is a step forward for the White House, and a nuanced move toward better transparency policy.  To move beyond the first two years, the Obama Administration has to be willing to force agencies to make meaningful distinctions about their work, and publicly force them if necessary.  This memo, and the disclosures it should create, can help transform how our government undertakes its duties.

President Obama recently attended an Expo on Democracy and Open Government on his trip to India and announced the creation of a US-India Dialogue on Open Government, a partnership that the White House blog declared to be "built on shared values." It is very exciting to see the President's renewed commitment to this issue, particularly with one of the world's great democracies and a country for which I have a personal fondness.

It is equally exciting to see that the President had an opportunity to meet with many of  the same Open Government groups that I personally visited less than a month ago, including the NGO Janaagraha, that focuses on empowering citizens and actively engaging with government for change. Sound familiar? They have built online tools on exciting ideas like their 'I Paid a Bribe' distributed research project to keep local officials accountable for illegally demanding brides. (Still trying to think of a way to adapt this for the U.S.)

After returning from my trip in India with my colleagues from Omidyar Network I wrote:

Sunlight’s work is certainly far more data centric than many of the organizations I visited. And I didn’t see anything that rivaled some of the tools for digesting and using that data that we’ve built.  But some of the university-based projects and other NGOs have figured out things we haven’t on the engagement front, including excellent online and offline organizing models around government accountability, and thinking way ahead of us on the mobile platform. And while certainly some of the problems are different (real bribery is still prevalent in many places as opposed to the “honest graft” we have in the US), there’s tremendous room for cross-continent collaboration.

I agree with what appears to be the President's take away. These are fascinating developments in India. As Samantha Power on the White House blog wrote:

India is at the vanguard of figuring out how to exploit technology and innovation on behalf of democratic accountability. U.S.-based groups, as well as those throughout the developed and developing world, could learn an enormous amount from these efforts.

Sunlight is happy to already be in dialog with many of the groups that had the opportunity to shake the president's hand in Mumbai. We do have the same goals. And as we continue to advocate for open government here , we encourage the President to lend more of his time and energy to the effort here at home. Perhaps the White House could start by hosting an expo of open government tools built by American organizations for American politics?

Update: Read some of the additional materials that the folks over at TechPreisdent have secured.

On Thursday, the White House announced the winners of their Leading Practices initiative, that they first outlined in April.

The Leading Practices were designed to highlight examples where agencies have risen above the expectations set by the White House, and proactively attained a higher standard of transparency. (I participated in helping to establish the leading practices standards.)

The winners are a collection of some of the best transparency work being done at federal agencies, with HHS taking the slot for transparency (quite deservingly). These winners are a nice counterpart to the White House page on Open Government Highlights.

As I wrote when the Leading Practices were first announced, though, there is a bittersweet element to this congratulatory platform. As the White House rightly points to the great work some agencies are undertaking, we can't help but wonder whether there is an analagous effort being undertaken with agencies who are struggling with (or blowing off) the Directive's requirements.

While we can hardly expect the White House or OMB to publicly chastise any laggard agencies, we do have to wonder how much of a private stick exists to go along with this public carrot.

Last Friday, the White House released a new batch of visitor logs covering last October, fulfilling a pledge they made last month. Over here at the Sunlight Labs, we took the logs and added them to the handy online, searchable database we created last month, so that you can see for yourself who is coming to the White House and why.

This is the first full month that has been release by the administration and adds almost 100,000 new records for October. As we mentioned back in January, this is a positive step by the Obama administration, and we are happy to see that they are committed to releasing this data in a timely basis.

We still don’t know how many records are being withheld, and for what purposes. It would be nice for the White House to release at least a number, and ultimately a justification (read: national security) for why those names have been redacted. None the less, this is still part of a much larger, unprecedented level of transparency on behalf of the administration.

One of the other problems with the White House visitor logs is that there is no real accurate way to ensure that if you see a “Samuel L. Jackson” in the logs, it's actually the actor. It could just be another Sam. That’s why we caution you, when you are reading through the records and doing your own independent research not to jump to conclusions. Otherwise, happy hunting!

Public appetite for transparent health care negotiations is driving us toward more transparency.

While Congress has rightly responded to that public pressure by posting bills online for 72 hours, the public dialog about the process of shaping health care legislation is more focused on transparent deliberations.

President Obama's promise to put healthcare negotiations on CSPAN, in combination with the question of formal conference proceedings, has become shorthand for several more fundamental questions. Formal requirements for public proceedings, while sometimes appropriate, are far short of what we should all be aiming for.

Firm requirements for public deliberations, since they are essentially prohibitions on private speech, are probably inappropriate for issues like health care in Congress. Since you can't require those negotiations to be public, Promises and Requirements are downgraded to Suggestions and Inducements.

Maybe that's why it's been so easy to ignore what's possible for health care deliberations. Since President Obama promised deliberations on CSPAN, shouldn't we all focus there? It's right to focus on a Presidential candidate's promises, and they should mean something. But if it turns out to be a failure, that doesn't mean we should all just go home. By focusing too much on requirements and promises, we're missing out on a chance to conceive and create public dialog that does work.

That's the realm that we shouldn't be ignoring. Where hard and fast requirements can't deliver what we're all looking for, we should focus on thinking of what can deliver it. "Put it on CSPAN!" should start to address those more basic questions.

We've all been so focused on the precise language of President Obama's campaign promise -- did he or didn't he keep it? -- that everyone was gobsmacked by last Friday's appearance before the House Republican retreat.

Since the appearance was almost universally welcomed, why haven't there been calls for just such an appearance? Because the public dialog has been focused on evaluating promises and procedures (the realm of the requirement, which is ultimately insufficient for public deliberations) rather than on what we actually want to see.

That's what was shocking to me about last Friday. Not just that the President and House Republicans were engaged in an honest, unscripted public debate, but that it wasn't orchestrated beforehand, or the direct result of public pressure.

House Republicans and President Obama innovated in the face of diffuse public pressure.

We live in a world where live streaming, immediate clippable archives, and all manner of new public interaction are now possible. We should balance our judgment of the world of public deliberations requirements (conference committees, or exaggerated promises) against the world of what is possible and desirable.

We should also remember that inducing public deliberations into the public sphere is exactly the point of much of our politics. Sunday talk shows, discharge petitions, Dear Colleague letters, caucus meetings, and editorials are all, in their own way, attempts to cajole, drag, and otherwise induce a policy conversation into the public eye.

Of course, our new technological capacity is having an effect on each of those spheres as well.

We advocate for a 72 hour rule precisely because it empowers all of us to take part in a more substantive, valuable public dialog. Government information empowers participation, and the Sunlight Foundation exists to empower the public through access to information.

As technology leads us to have higher expectations, and politicians are forced to respond with new methods for including the public, we should all respond with better expectations about the possible and the desirable.

If we don't, we'll become more susceptible to fake public engagement, and lose the chance for new technology to lead to a better relationship between citizens and their representatives.

President Obama's weekly address explained on his administration's efforts to combat influence peddling, and the steps it is considering taking in response to the Citizens United decision. It will be interesting to see to what extent these themes are reflected in the State of the Union speech this Wednesday, and how they translate into policy. The Sunlight Foundation will, of course, remain focused on the transparency implications.

Some highlights from the weekly address are after the jump.

First, the President described his lobbying reform efforts over the past year:

On my first day in office, we closed the revolving door between lobbying firms and the government so that no one in my administration would make decisions based on the interests of former or future employers. We barred gifts from federal lobbyists to executive branch officials. We imposed tough restrictions to prevent funds for our recovery from lining the pockets of the well-connected, instead of creating jobs for Americans. And for the first time in history, we have publicly disclosed the names of lobbyists and non-lobbyists alike who visit the White House every day, so that you know what’s going on in the White House – the people’s house.

He then talked about the effects of the Citizens United decision.

This [Citizens United] ruling opens the floodgates for an unlimited amount of special interest money into our democracy. It gives the special interest lobbyists new leverage to spend millions on advertising to persuade elected officials to vote their way – or to punish those who don’t. That means that any public servant who has the courage to stand up to the special interests and stand up for the American people can find himself or herself under assault come election time. Even foreign corporations may now get into the act.

And finally, he indicated that responding to Citizens United is a priority for the administration.

When this ruling came down, I instructed my administration to get to work immediately with Members of Congress willing to fight for the American people to develop a forceful, bipartisan response to this decision. We have begun that work, and it will be a priority for us until we repair the damage that has been done.

I was out of town earlier this week when the Open Government was released and so I am just now weighing in with a few thoughts. Without being too over the top (OK maybe I am) I think this Directive potentially represents a watershed moment for democracy, the likes of which can forever change the relationship between the government and the public it serves.

This Directive acknowledges that the Internet is the right, proper and primary medium for communicating information to the public, and that "the public" is key in helping determine the policies, directions and priorities for government's work. In doing this, the Directive requires each executive agency to create an online portal designed to promote important agency data and provide a place for citizens and agency officials to work together.

Every citizen --  from web developers to journalists to a real estate agent in Kentucky or mother in Colorado -- has something to gain from this new initiative. If you care about the health of your children, the safety of your workplace, the crime in your streets, or corporate accountability you will have new information to inform yourself and discuss with those who represent you.

Business will have a lot of new data to base their decisions on. Real estate agents and developers will be able to see migration patterns by income when the Internal Revenue Service releases their database of tax-filer migration from county-to-county or state-to-state. This will aid in decisions on where to build, what kind of market to expect and what type of people live where. Similarly, data released by the Department of Housing and Urban Development on public housing will not only help reveal slum lords, but also help renters or buyers locate Section 8 density when deciding on a neighborhood to move into.

Here are some more examples:

•    The Federal Aviation Administration (FAA) plans to release a real-time online database providing up to date information on flight delays and cancellations. This database will take existing FAA data and combine it with data from the National Oceanic and Atmospheric Administration (NOAA) to allow the public to be able to easily access online the status and causes of airport delays and cancellations. •    The Department of Homeland Security (DHS) is creating Virtual USA, an information-sharing system that will reduce lag time in responses to emergency situations. This system will likely save lives and empower local authorities with better information to reduce the costs of emergencies and disasters. •    The Department of Veteran's Affairs (VA) will release raw data and “report cards” on Veteran's Administration hospitals empowering current and former military service members and their families to know the quality of care that they are getting when they go to a given VA hospital.

The Directive will also help citizens hold their government accountable through new avenues of government disclosure and citizen engagement. For example, the Department of Justice is collecting and publishing agency reports on their Freedom of Information Act (FOIA) compliance. Each agency report will include information on how long it takes to process FOIA requests and which agencies are most successful at fulfilling FOIA requests from the public (and consequently which agencies are not as well).

The Directive also requires each agency to allow citizens to request data to be released by agencies and placed online. This process will enable citizens to continue to press officials to make their data as open as possible, and that call for more information will have an affect far beyond the executive branch.

The General Services Administration (GSA) will be releasing a full database of all federal advisory committee members that can be mashed up with lobbying records and contribution databases to show the influence that resides on these important bodies - and we can similarly expect Congress, states and municipal governments nationwide to feel pressure to release information. That is, if the public demands it.

It will be our responsibility as citizens to monitor the data quality reported by agencies and the timeliness of the reporting requirements. Just like all previous open government moments, this one will require constant public engagement to ensure that will be all that it can be. Sunlight is bullish on this Directive because we believe it will make permanent the idea that open government means an online government. In the digital age how can it mean anything else?

Even if you don't realize it yet, the plan laid out in the Directive will impact you, so we hope you'll join us during this exciting to begin a this new dialogue with government. What we have now is a plan. And it is a plan that will require public engagement to ensure that the policies it lays out are enacted and undertaken to make our dream of an open government a reality.

As promised by the White House, new stimulus lobbying policies have been released by OMB.

The new guidance clarifies restrictions on lobbying, lifting a more general limitation on meetings between federally registered lobbyists and administration officials, and clamping down on all communications regarding awards for carefully tailored time periods.

We're particularly excited about this announcement:

A web tool is being developed to facilitate disclosure of lobbyist contacts concerning the Recovery Act, and the tool will be available shortly for your agency’s use.

...given that we've long advocated for better real-time disclosure of lobbying contacts online, going so far as to mockup what such a system might look like.

I've embedded the document below. H/t Ellen's and Gabriela's twitter feeds.

As ProPublica just pointed out on Twitter, the White House today released staff salaries, as they do each year.

Since 1995, the White House has been required to deliver a report to Congress listing the title and salary of every White House Office employee.

What's unique about this year's report is the data that accompanies it, helpfully presented in a sortable chart, which is itself downloadable.

Using Google Spreadsheets, I made the following chart, to see whether my first impression was correct: that many White House staffers were concentrated at either $30 - 40,000 per year, or around $160,000. Turns out my first impression was wrong, and that there's a fairly even distribution.

This is a step in the right direction, where a simple explanation, accessible chart, and exposed data all combine to give a useful look at something that's required to be made public.

(Note -- please interpret this chart as a stab at making sense by an amateur, and should not represent Sunlight's visualization work, which far, far exceeds my own.)

Today is 120 days after President Obama's Open Government Memo, released on his first full day in office: January 21st, 2009.

In case you're wondering what's formally due today, or whether anything is late, here's what the memo requested:

I direct the Chief Technology Officer, in coordination with the Director of the Office of Management and Budget (OMB) and the Administrator of General Services, to coordinate the development by appropriate executive departments and agencies, within 120 days, of recommendations for an Open Government Directive, to be issued by the Director of OMB, that instructs executive departments and agencies to take specific actions implementing the principles set forth in this memorandum. The independent agencies should comply with the Open Government Directive.

The Directive itself isn't due today, as has been widely suggested.