On Wednesday, White House press secretary Josh Earnest sent The New York Times a letter asking journalists to give the Obama administration credit for government transparency, arguing that media columnist Jim Rutenberg did “did not acknowledge the important and unprecedented steps that the Obama administration has taken to fulfill the president’s promise to lead the most transparent White House in history” in his latest column.

Since January 2009, the federal government has become more transparent in some ways and far less so in others, which naturally leads the press and public that relies on those assessments to view the success of President Barack Obama’s pledge to have the “most transparent administration in history.”

In June, Brian Stelter, the host of CNN’s Reliable Sources, suggested that this is the least transparent administration in history, pointing to limited access to the White House, official secrecy and over-classification, and record numbers of denials to Freedom of Information Act (FOIA) requests for agencies, much as Rutenberg did in his column. The Washington Post’s Margaret Sullivan hit the same points in a recent column on the administration’s record. On Wednesday, Andrew Beaujon critiqued Earnest’s letter, calling attention to the full record. Erik Wemple took an inventory of journalists who disagreed with the White House press secretary’s assessment, drawing from the Committee to Project Journalists’ scathing 2013 report on the Obama administration and the press.

In his letter, Earnest asked: “If journalists don’t acknowledge steps that the Obama administration has taken to strengthen transparency, then who will?”

We’d ask him to look beyond the press corps in the briefing room and cable news shows. The White House press secretary ignored the role that government watchdogs have played in evaluating the strengths and weaknesses of the Obama administration’s open government initiatives, from the Project on Government Oversight to OpenTheGov.org to Public Citizen to the Government Accountability Project to Sunlight.

Over the past seven years, we have both acknowledged this administration’s progress on open government, decried its failures and advocated for reform. The challenge that we and the rest of the country face in judging the veracity of Earnest’s argument is that there is no consensus for what the “most transparent administration” in U.S. history is, nor how we should assess it.

In that context, what would be the right gauge for judging adherence to transparency and open government? Here are some approaches to evaluating an administration’s openness that go beyond the headlines, and some thoughts about how to sort through them.

—–

Compliance with FOIA

How the federal government handles public records requests and lawsuits is often used as a litmus test on transparency because it’s easy to measure, although the Obama administration has made claims about releases that independent analyses of its own data may not support.

The challenge with using FOIA compliance as a historical measure is that the FOIA only goes back 50 years, so it’s impossible to compare any administration before President Lyndon Johnson’s on this count. In 2015, there were a record numbers of requests — and a record number of denials. It’s not at all clear that presidential guidance, even executive orders that carry the force of law, have significantly affected agency performance or even regulations, as recently shown by the Department of Defense.

On top of that, the baseline number of FOIA requests and their content is not a constant, further complicating evaluation. That said, it is useful to compare the policy of disclosure regarding the “presumption of openness” issued in the Obama administration to that of the last administration, in which a U.S. attorney general fundamentally undermined the purpose of the law. It’s also useful to consider secret opposition from agencies, financial interests and the Justice Department itself, as we did in our assessment of the watered down FOIA reform bill that passed Congress and made it to the president’s pen.

Media access to the president

Healthy tensions between the press and presidency go back to the nation’s founding, with the appropriate balance between access and official secrecy shifting over time.

In a letter to Edward Carrington in 1787, Thomas Jefferson hailed the importance of newspapers to our union. Teddy Roosevelt famously invited many journalists into the White House to debate the issues of the day, although that decreased over the course of his administration.

Despite shifts in technology that have enabled the White House to go direct to the public over the internet, social media and smartphone apps, the White House press corps still plays an essential role in holding the power of the presidency accountable.

The record of the press and Obama presidency on this count has been well documented, including protests regarding access. The Society of Professional Journalists has pushed for greater transparency and relaxed rules on access.

Members of the media who cover the administration daily argue that the access of the White House press corps to senior officials and President Obama, including meetings with business titans or foreign leaders, is a good litmus test for transparency in the federal government writ large. It’s certainly a proxy for a president’s commitment to transparency in their White House.

While he has sat down for hundreds of interviews, the president has not, to our knowledge, talked to the Washington Post or New York Times chief White House correspondents on the record. In doing so, President Obama didn’t just fall short of his pledge to be more transparent: The public hasn’t gained valuable insight into many of his decisions in office. Jeffrey Goldberg’s thoughtful, comprehensive assessment of President Obama’s foreign policy philosophy and record for The Atlantic shows exactly how such access can be applied to public interest journalism.

Number of press conferences

The number and quality of press conferences a president holds in office affirms the fundamental role that journalists play in our democracy. President Obama has given an average of 20 press conferences every year, comparable to the last three presidents — ahead of Nixon and Reagan, but far behind Franklin D. Roosevelt or Calvin Coolidge.

Public access to government scientists and research

The quality and ease of media access to government scientists is a critical aspect of informing the public about scientific research and validation of their rights to freedom of expression.

The Obama administration has been criticized for restrictions and filtering, though this has reportedly improved in some agencies. A 2015 report on mediated access from the Union of Concerned Scientists and the Society of Professional Journalists highlighted the ongoing challenges here.

[W]hile much progress has been made, policies and practices regarding media access and scientists’ freedom to communicate vary widely across agencies. Our updated 2015 report, Grading Government Transparency, gives a detailed analysis and letter grade to each agency’s media policy. However, writing a strong policy is only half the battle; it must then be implemented effectively. Some of the agencies with the strongest media policies come up frequently when journalists describe difficulties they have encountered getting access to scientists.

In that context, how should we weigh the addition of a public information officer to interviews with government scientists? Or policies that make federally funded scientific or social research open and accessible to the public online?

Support of open government reforms moving through Congress

Whether a president sends draft laws to Congress, supports bills or opposes legislation is also a proxy for their commitment to transparency and accountability. The Obama administration never officially commented on FOIA reform in 2014, while the Department of Justice secretly lobbied against it. While we celebrated President Obama signing FOIA into law, we also noted the absence of a legislative proposal from the White House that would codify the administration’s FOIA policy into law and the absence of support for bills that used the exact same language of those policies. The administration’s lack of support for the DATA Act was similarly notable — symbolized by a private signing, despite overwhelming bipartisan support for it in both houses of Congress. So, too, is the current silence of the White House regarding an element of the National Defense Authorization Act, which would effectively exempt the Department of Defense from the FOIA.

Support for whistleblowers

Given the fundamental role that whistleblowers play in highlighting fraud, waste and abuse, an administration’s record upon the incidence, manner and prosecution of whistleblowers (or policies that protect or expose them) is a critical part of its open government record. The way a White House takes legal or administrative action regarding leaks that demonstrate unconstitutional behavior in the executive branch matters.

On the one hand, the Obama administration added to congressional legislation with more protections for whistleblowers through an executive order, save for an exemption for those working in or contracting for the intelligence agencies. On the other, there have been a record number of cases filed against whistleblowers under the Espionage Act, more than all previous administrations combined. The double standard for “senior administration officials” quoted in stories favorable to the White House and whistleblowers who go to the press with embarrassing facts is one that no future administration should espouse.

Protection of domestic press freedom

The secret surveillance or prosecution of journalists for protecting sources is a direct threat to the ability of the press to document and expose government corruption and inform the public about what is being done in their name with tax dollars. There was considerable, understandable furor of about the U.S. government seizing the phone records of a member of the Associated Press in a search for the leaker of a confidential initiative in the Middle East, a practice which the Department of Justice acknowledged and then reformed.

Open data and proactive disclosure

Given the hugely expanded role that the internet and associated technologies play in our daily lives, opening up data and disclosing it online has taken on outsize importance. As in other areas, however, context matters: Open government data releases that relate to economic activity or service delivery or science matter, but they’re not a replacement for disclosures of how power is being wielded, from the use of force to regulatory decisions or calendars and correspondence. While Data.gov has improved, as has the quality and quantity of data on it, concerns persist in the journalism world about the prevalence of HTML and PDFs on the site that are not responsive to public records requests.

As Sunlight has called for since its inception, the Obama administration has published huge amounts of open government data online in almost every sector the federal government oversees or influences, modernizing how the federal government uses the internet to communicate and collaborate with the public and creating new platforms and formats for disclosures of consumer complaints, corporate fees or environmental science, including weather and climate data. In July, the Obama administration published long-awaited reforms to the federal government’s information policy, many of which are important updates to how our government collects, uses and protects public assets.

Civic engagement and open innovation

Around the world, governments are increasingly arguing that a broader set reforms be considered under the rubric of open government, from traditional e-government to social media. To the extent that “crowdsourcing” efforts aimed at solving grand national challenges or involving more of the public in citizen science engage and inform citizens, this is useful. Similarly, using platforms like regulations.gov and online comment systems to provide more ways for a distributed population to be involved in rule-making and provide feedback, as with the Federal Communication Commission’s Open Internet Order, creates meaningful new avenues for public participation and knowledge about governance processes. Releasing public comments on such proposals for analysis enables a healthy discussion regarding who is participating and evidence for regulatory capture. Publishing open source software code and creating an official source code policy in the open both take advantage of new horizons. These new points of collaborate also provide opportunities for collaboration with allies and can be used to mask secrecy in how power or influence is actually being wielded.

Sunshine laws

Evaluating claims to transparency should also include how well a given administration complies with sunshine laws, including the Federal Election Campaign Act passed in 1971, the Government in the Sunshine Act in 1971 and Ethics in Government Act in 1978. On that count, the relaunch of the Federal Register online significantly improved information about what government is doing or proposing, which is core not only to transparency and accountability but to achieving the consent of the governed.

National security and classification

Investment and relative progress on declassification and official secrecy are also relevant. So, too, is considering the impact of the initiatives like the Obama administration’s “Insider Threat Program” on openness, or years of secrecy around the use of drones for unprecedented targeted killings in counterterrorism. The disclosure of civilian casualties from such strikes has been long in coming.

Given the global significance of mass surveillance, the absence of proactive disclosure of mass surveillance using bulk telephony or the Internet backbone is notable. How should we weight secret legal interpretations of secret laws that govern such surveillance? How should we consider the Obama administration’s released and undisclosed legal opinions, memoranda and other documents on both counts — in the wake of the leaks by Edward Snowden — against that record?

Corruption, influence and scandals in the White House

The incidences of major corruption, fraud or malfeasance by officials or Cabinet members, or their frequency, or their investigation and consequences matter to the record as well. In modern times, we could cite the use of personal email accounts for government business by Cabinet members, or the destruction of millions of emails in a White House.

The recent 50th anniversary of FOIA’s passage provided an opportunity to remember why the nation’s most important open records law was passed in the first place: the aftermath of the Watergate scandal, which led to the resignation of President Richard Nixon. While Watergate looms large in the consciousness of the public and press, many administrations have faced scandals that led to the resignation of Cabinet secretaries, from President Jimmy Carter’s director of the Office of Management and Budget to President Zachary Taylor’s secretary of the Department of War.

Influence and the disclosure also matters. What should we make of the publication of White House visitor logs online for the first time, which have been used and cited by many members of the media as evidence of influence? Ellen Miller, the founding director of Sunlight, hailed the shift as a “huge step toward a more transparent government” in 2009; but as she observed then, the watershed took place because of a lawsuit, which is context Earnest neglected to include in his letter. Or how should we value Ethics.gov? Or President Obama’s embrace of super PACs and the deregulation of money in politics?

Promotion of open governance globally

Whether a given president comments on open government or its importance, much less advances an affirmative policy for his or her administration, also matters.

While we can and should debate about the success, failure or coherence of the Open Government Initiative, there’s no doubt that President Obama issued a memorandum on his first day of his office, followed up with an Open Government Directive and then an open data executive order in 2013. It’s also essential to acknowledge that his administration launched a global Open Government Partnership, a multilateral, transnational effort that has created a platform for civil society to hold the governments of participating nations to account, although the effort received relatively little attention in media. Regardless of the administration’s performance, openness has certainly remained a part of Obama’s stated ambitions, and a core part of his conception of democracy.

—–

A mixed record

Taken together, one might achieve some kind of useful determination about how the Obama administration compares, even if it’s hard to speak to FOIA since it’s only existed for 50 years.

On each count, answers exist that contradict the Obama administration’s “most transparent ever” claims — but context matters. On many ways, this administration not only compares well with history but has made history. On others, not so much. If you see metrics that we have not outlined here that should be considered, please suggest them.

This administration has made unprecedented releases of data with a measurable impact on many sectors of society at the same time that the White House and agencies have stonewalled the press asking tough questions. When it comes to the use of the internet to disclose data and documents, there are only two prior administrations that can be relevant.

President Obama’s choice to make open government part of his first day in office reflects an optimistic worldview that connected democracy and legitimacy to transparency and accountability in the wake of a decade of secrecy around an expanding national security state.

While the actions or inactions of his administration contradicted this in deed, it would be an error to dismiss the ideas themselves or the investments in time, money and technology as meaningless or divorced from an important aspect of American democracy that this presidency has deepened.

The most definitive statement we can make today is that President Obama will leave office with a mixed record on open government, although it’s not too late to embrace reforms that will cement the aspects of this legacy that have merit or to connect these initiatives to the ideals and principles that the American experiment was founded upon.

We hope that he and his staff do so, in public, ensuring that whoever occupies the Oval Office can not only build upon the successes but learn from the failures. Public trust in our government depends on it.