When the Sunlight Foundation received an inquiry from the Central Intelligence Agency last week, we weren’t sure what to expect, given the recent pace of world events. The news turned out to be straightforward: the CIA was going to publish approximately 12 million declassified pages from its CIA Records Search Tool (CREST) on the Internet.

This afternoon, the CIA carried through on its commitment from October 2016, making nearly a million individual archived documents available to the public online in its Freedom of Information Act reading room.

The CREST collection goes back to the 1940s and the origins of the CIA, covering the Cold War, the Vietnam and Korean wars, the Berlin Tunnel project, aerial reconnaissance, and more. There’s even a section on a STARGATE project, which might lead to renewed speculation about what our federal government knows about extraterrestrial life.

It’s important to emphasize that these documents aren’t new: they’ve been available to researchers at the National Archives in College Park, Maryland. If this corpus of documents represented “the CIA’s secret history,” they haven’t done a particularly good job of keeping it that way over the past 17 years or so since the CREST tool first became available in 1999.

The records in question have been declassified due to the provisions of Executive Order 13256, which used to be 12958, issued under President Clinton in 1995. The order required “the declassification of non-exempt historically valuable records 25 years or older.” If agency staff decided that records fell under an exemption of the Freedom of Information Act, in other words, you’re not going to find them online unless someone sued them out.

The shift today is that the CIA has used the Internet to make the declassified files available to all of the public, wherever we go online. That’s not a minor shift: the impact of open government upon public knowledge and trust is predicated upon access. Declassifying millions of documents doesn’t inform anyone if they just sit in a dusty file cabinet.

Given Sunlight’s decade of advocacy for more open government through technology, we took some time today to talk to the CIA’s director of information management, Joseph Lambert, a 32-year veteran of the civil service, about why the agency was broadening access beyond the walls of four computers in NARA.

“The CIA is made up of American citizens just like you,” he told Sunlight, over the phone. “The people that I work with, we believe that we hold these records in trust for the American people. When their sensitivity attenuates over time, we feel we have a responsibility so the American people can judge them for themselves. It’s important that we put these source documents online.”

When asked about which documents would be of the greatest public interest, Lambert noted that the materials from the Berlin Tunnel get a lot of attention in College Park. (The CIA knows this because they log file access and printing.) He also highlighted science and technology research and development files, reports from operations in the middle of the 20th century, and materials on secret writing and invisible ink.

In the 21st century, we’ll now be able to see if public fascination with these aspects of spycraft endures, should the agency participate in the federal government’s Web analytics program.

As more documents are declassified, the public should expect more of them to flow onto this reading room, along with other materials responsive to Freedom of Information Act requests.

When asked how the agency was approaching declassification, Lambert said that they’ve been working on this over the past eight years.

“We are focused on improving transparency and releasing what we can,” he went on. “We involve experts. The standard is damage to national security. We have classification and declassification guides that will guide if there will be damage. That is really the impetus for what makes it out the door and what does.”

When asked if President Barack Obama’s Open Government Directive had an impact upon this work, Lambert said that it had, noting that he was involved in writing the CIA’s first open government plan, in 2010. (As we reported earlier this month, however, the CIA has not published a new open government plan since. When we called this to the CIA’s attention, Lambert said that they were “in the process of updating it now” and would follow up. We’ll note it if and when it happens.)

“The focus on open government and transparency has had positive effects,” said Lambert. “We’ve had 9 declassification events. We’ve partnered with presidential libraries and major universities, and looked at our archives to see what compelling stories were there and if sensitivity had been attenuated. We’ve told stories that positive to CIA, told others where got wrong, like the Korean War. We’ve tried to get a body of work out there where American public can judge for themselves.”

The challenges the agency has faced in its declassification efforts in the past, however, pale in comparison to what lies ahead, as the pace and scale of data and document creation increases.

“When I was starting my job, about 2 million pages passed through my office every year,” said Lambert. “Now, it’s about 12 million pages. We are going to have to scale from tens of millions of pages to hundreds of millions of pages. We can’t do that with just people. I did the math: we would need 2.5 million people in one of my 3 divisions. We can’t just deputize all of Fairfax.”

Lambert told us that the agency will be focusing on machine learning and natural language processing software to help them, bring technology to bear.

“We have spent time with the Archivist of the United States and the White House on automating these efforts,” he said.

What the agency’s public relations efforts left out, however, is that the public can also thank MuckRock, a nonprofit that helps people to file Freedom of Information Act requests, for today’s transparency watershed. (Sunlight provided a grant to help MuckRock started, years ago. Our investment has been more than returned by the public knowledge they have created since.)

As Jason Leopold reported, MuckRock filed a lawsuit in December 2014 to gain access to the entire CREST database.

“The CIA told MuckRock it would take at least six years to release all of the documents,” noted Leopold. “Frustrated, Michael Best, a journalist and researcher, launched a Kickstarter campaign to raise funds to manually copy and scan all of the documents.”

As it turned out, it took just over two years. (Here’s hoping the agency figures out how to accept FOIA requests over the Internet using a new FOIA.gov even faster, dumps that fax machine in the dustbin of history, and follows up on all outstanding FOIA requests.)

While we’re not thrilled about the fact all of these documents have been published as PDFs, today is another step in the ultra-marathon that is open government in the United States. Progress is progress, and should be celebrated.