Explained: The shutdown of the National Guideline Clearinghouse and the independent efforts to launch a replacement

by

On Monday, July 16, the Agency for Healthcare Research and Quality (AHRQ) shut down its National Guideline Clearinghouse (NGC), formerly hosted at www.guideline.gov, a website that had gotten about 200,000 visitors per month, according to AHRQ, and, for almost 20 years, had been medical professionals’ go-to resource for finding and understanding medical guidelines.

After the Web Integrity Project’s article about the planned NGC closure, by our investigator, Jon Campbell, was published in The Daily Beast, voices from the worlds of medicine, civic tech, and politics rose up to oppose the site’s shuttering. Efforts by members of the public to set up a public-facing archive of the NGC’s content started almost immediately. Separately, non-profits that specialize in work with medical guidelines and have been focused on the shutdown for months have announced plans to independently launch their own guideline clearinghouse websites. But, so far, none of the initiatives announced will be able to serve as a replacement to the curated, easily-searchable NGC website, whose database AHRQ will be retaining as they search to build a public-private partnership to relaunch the site.

For everyone concerned about the NGC shutdown but struggling to follow the explosion of attention it’s gotten in the last week, we’ve put together an explainer of the government’s plans for the website and the key ongoing efforts to stand up a replacement:

What happened to the NGC and what are AHRQ’s plans for its content?

While it might be fair to say that the NGC was “deleted,” it’s probably clearer to say that the website was shut down or removed from the public domain on July 16. AHRQ first announced the shutdown in April.

The NGC’s content is no longer available at the URL that hosted the NGC, www.guideline.gov, or on any other live federal government website. The former URL now redirects to https://www.ahrq.gov/gam/index.html, an informational page about AHRQ’s now-terminated guideline clearinghouse operations. Our recent Web Integrity Project report also demonstrates that the federal government’s archives of the NGC are far from complete, lacking its search functionality and a significant portion of its content.

But it’s important to note: The database and all of the content that underlied the NGC is not being deleted from government hard drives; AHRQ is retaining it, as required by federal records law, and announced that it’s “exploring options to support the NGC in the future,” perhaps as part of a partnership with the public- or private-sector, although it hasn’t provided more details about how it might hand off the database to launch a replacement for the NGC.

AHRQ certainly can’t take credit, however, for any ongoing efforts to build archives or launch independent websites listed below ― in terms of evaluating its stewardship of the NGC database, it hasn’t resolved anything thus far.

What original content did the NGC host and what will its absence mean?

The NGC compiled and curated guidelines from hundreds of medical organizations, a group comprised predominantly of medical specialty societies, professional associations, and governmental medical agencies, both inside and outside the US.

Though the NGC did not directly host medical guidelines themselves, which are typically copyrighted material, it commonly linked to the source guidelines on medical journal or society websites. Instead, the NGC was known for its guideline summaries, which pull the key recommendations, and ratings for the strength and the quality of the scientific evidence for each recommendation, out of guidelines that extensively cite scientific research and are sometimes over a hundred pages long.

As of last year, the NGC also began providing new assessments, evaluating the degree to which each guideline follows the National Academy of Medicine’s (formerly the Institute of Medicine) standards for trustworthy evidence-based clinical guidelines. The work to produce these guideline assessments, which have only been completed for about 150 of the NGC’s guideline summaries, has also been terminated with the NGC closure.

According to Karen Schoelles, the director of the Evidence-based Practice Center at the ECRI Institute, the non-profit organization that had served as AHRQ’s contractor to create and maintain the NGC for almost 20 years, the new assessments would’ve given physicians more confidence in the guideline developer’s ratings. She explained that, “under the contract with AHRQ, we were primarily looking to see if guideline developers followed appropriate procedures to come up with the ratings they gave their recommendations, instead of doing the rating ourselves and checking their accuracy.”

Risha Gidwani-Marszowski, a health economist at the VA Health Economics Resource Center and an assistant professor at Stanford University, was one of the experts who spoke up to support the NGC and its mission in the past week. She told me that the NGC was important in facilitating the impact of medical research on medical practice. “The state of medical evidence today, much like that of the rest of the digital world, is one of information overload. Having a trusted source free of financial interests condense and evaluate the quality of medical guidelines is key to ensuring medical evidence appropriately influences medical practice.”

Search is essential: The value of the NGC also came from it being more than just a repository; its list of guidelines was indexed by an attribute template to provide users sophisticated search functionality across a comprehensive classification scheme, including the ability to search by guideline rating. A “Matrix Tool” widget even allowed users to search by creating tables cross-referencing all pairs of two classifications of attributes. Gidwani-Marszowski said, of the NGC’s search capabilities, that, “the utility was not simply a static presentation of guidelines summaries; real and unique benefit is to be found from features that allow for filtering and comparison of summaries.”

The NGC’s clear inclusion criteria for guidelines also meant it could play the role of helping medical professionals determine which guidelines are trustworthy. Valerie King, a professor in the Department of Family Medicine and Director of Research at the Center for Evidence-based Policy at Oregon Health & Science University, who first spoke with WIP’s investigator for The Daily Beast article, touted the NGC’s “gatekeeping” as a critical role that would be missed after its closure. “In times past, there were an awful lot of, let me put air quotes around this — ‘guidelines’ — that weren’t of good methodologic quality,” King said. “They were typically just expert opinions, or what we jokingly refer to as BOGSAT guidelines: ‘bunch of guys sitting around a table’ guidelines.”

How do the announced independent non-profit replacement initiatives compare to what the NGC provided?

While important efforts to continue the work of the NGC and launch a website with comparable content have been announced, there are no organizations, thus far, that have stated publicly that they have plans to launch a genuine NGC replacement.

On Tuesday, the ECRI Institute, AHRQ’s non-profit NGC contractor, announced that it would be continuing its work and launching an interim website this fall to “provide a centralized repository of current, properly vetted evidence-based clinical practice guideline summaries and other information.”

Unfortunately, according to Schoelles, AHRQ holds the copyright on all content, including the guideline summaries and rating assessments, that ECRI produced under the contract. Schoelles said that they would like to be able to use the content as published on the NGC site for the replacement site they’re launching. “It would certainly speed things up if we had that kind of opportunity ― we’ve spoken to AHRQ about that, and we’re still waiting for an answer.”

That means that, because of AHRQ’s silence, ECRI will essentially have to start their work of summarizing guidelines from scratch. Schoelles said, “We’re going to be reworking the guideline summaries, and we will inevitably have to do some duplication of efforts we’ve done previously.” In the meantime, ECRI is speaking with guideline developers to get permissions before their launch. While they aren’t licensing guidelines or any copyrighted material from guideline developers, ECRI’s practice is to get permission to provide guidelines’ recommendation statements and post their own guideline evaluations.

ECRI also likely won’t be able to quickly reproduce the search functionality that made the NGC’s guideline summaries so accessible to physicians and members of the public, and it remains to be seen what functionality will be available in ECRI’s planned fall website launch. This is because, without obtaining the database that underlied the NGC from AHRQ, ECRI will also have to reconstruct the NGC’s guideline classification and indexing scheme. But Schoelles is hopeful that, in working toward their replacement, they’ll also be able to make improvements, modifying the search functionality and altering how summaries are presented, to respond to feedback that they’ve gotten over the years.

ECRI is also, unfortunately, expecting that they’ll need to set up some sort of fee or subscription model, though they haven’t worked out the details. According to Schoelles, any fee will only be in place to recoup the costs of building a new website and the ongoing labor associated with continuing the work their staff has done to curate, summarize, and rate guidelines.

A paywall will be a huge blow to the accessibility of the guideline summaries and assessments. But, if AHRQ continues to keep the NGC database under wraps, and if the federal government fails to provide any other source of funding, there isn’t currently a clear alternative for how to continue the ongoing work of maintaining an up-to-date guideline clearinghouse.

What are the copyright considerations relating to guidelines and the NGC’s guideline summaries?

Because AHRQ holds the copyright on all content that ECRI produced for the NGC, including the guideline summaries and rating assessments, ECRI doesn’t feel its able to just repost the content when they launch their website.

But even in creating new guideline summaries, Schoelles explained that ECRI was newly seeking permission from the long list of guideline developers because the permissions they’d obtained previously, as AHRQ’s contractors, were for summaries that would be posted on the NGC, not on ECRI’s own website.

Jessica Fjeld, a lecturer at Harvard Law School and the acting assistant director of Harvard’s Cyberlaw Clinic, explained to me why ECRI’s reasoning makes sense. She said that, when posting new guideline summaries, “even if they think they have a pretty good fair use claim, defending a copyright infringement case can be costly, even if you win in the end. For that reason some organizations might want to seek permission anyway, particularly if they’re risk averse and they don’t want to worry about the uncertainty of facing a lawsuit.”

But Fjeld also noted, of copyrighted works, that “the actual language is what’s copyrighted, not the ideas behind it.” That’s likely why AHRQ feels comfortable that they’re able to hold the copyright on the guideline summaries and that they didn’t need to license the guidelines from the guideline developers in the first place.

That might mean that organizations like ECRI really could simply move forward with posting the exact same content that was on the NGC. Fjeld said that, “because AHRQ didn’t seek licenses for its summaries of the guidelines, other entities could publish the summaries if AHRQ allowed them to, by either relinquishing their copyright or making the summaries available under a Creative Commons license.” She clarified that, “that’s true assuming that the guideline developing organizations maintain the same stance and don’t newly object to their guidelines being summarized in another forum.”

Which portions of the NGC’s original content are currently available elsewhere?

GuidelineCentral has announced that almost all of NGC’s guideline summary repository is still available for free on their website. This effort, one of several that have publicly hosted the NGC’s guideline repository for years, is encouraging, and ensures that the removed NGC content will not become inaccessible to the public.

Its repository, however, along with those of other archival initiatives described below that have arisen in the last two weeks, currently provides less functionality than the NGC did for searching for and comparing guideline summaries. They, and others, may also continue work to reproduce the NGC’s classification and indexing scheme so that their resource can better serve as an NGC replacement.

Their repository will also only stand as a static archive of the guideline summaries and won’t be updated as new guidelines are published or with new assessments of guideline ratings.

Because AHRQ retains copyright control of the content that was available on the NGC, it’s also possible that they could ask organizations with NGC repositories, like GuidelineCentral, to remove their content from their website. But, again, if AHRQ gave approval or made the NGC content available under a Creative Commons license, GuidelineCentral and others would be in the clear and could continue providing their public repositories.

What role will the public archiving efforts of the NGC content play?

In the days before the NGC’s closure on July 16, members of the public and civic tech activists, including a number of “alt-gov” accounts on Twitter, began efforts to use Web scraping tools in order to store the NGC’s content. Their goal was to pull down as much of the NGC’s data as possible before the website went offline, to create archives that could then be deployed to build an online, public-facing portal to access the NGC’s former resources.

Fred Trotter, founder and CTO of CareSet Systems, which works with health data, launched one of the NGC repositories that’s gone live in the last week and explained his process for conducting the emergency backup here.

An archive alone can’t replicate the NGC: Unfortunately, because the back end of the NGC database, which stored the indexing data for the NGC’s guidelines, was not fully accessible through the website’s public-facing interface, efforts to launch a replacement website using these types of archives may fall short. While there are still many members of the public working hard to make their archives usable, thus far, no website has gone live that replicates the NGC’s functionality, in terms of guideline indexing or searchability.

And, as is true for the repositories mentioned above, even a fully functioning mirror of the NGC would not remain up-to-date, with new guidelines summaries and rating assessments.

Schoelles, from the ECRI Institute, explained that, “the upkeep is altogether different.” “A lot of the work is based on our relationship with the guideline developers, and we really delve deep. I have a team of people who work on the evidence reviews and help inform people who are actually writing guidelines ― we know a lot about the methodology and that gives us a whole different perspective.”

Archives and mirrors of the NGC may also be subject to copyright claims by AHRQ, if AHRQ does not give approval for the NGC’s content to be used and posted publicly.

How will the NGC shutdown affect the future use of medical guidelines?

Gidwani-Marszowski, from Stanford, made clear how damaging she thought the NGC shutdown was. “Its shuttering represents a significant loss for the medical community and to the practice of patient care,” she warned. “There is no other site today doing what National Guideline Clearinghouse did in presenting high-quality medical evidence to doctors and to the public.”

While ECRI plans to launch their replacement by the fall, that will mean the NGC’s former 200,000 visitors per month won’t have a place to easily search for and understand medical guidelines for months.

Schoelles said ECRI hopes that its replacement website can, eventually, serve medical professionals even better that the NGC did. “If guideline developers are willing to work with us on it, we’d like to work with them to develop clinical pathways and other tools, to better enable physicians to make decisions as new information comes in.” She also explained that, “if we can translate the guidelines and summaries into a machine readable format, then they may be able to be used in real time, not after the patient has gone home.”

“A number of people are developing fairly sophisticated applications that can interface with electronic health records and some of these might be able to help point physicians to decision support based on the most trustworthy guidelines during the course of their work.” Schoelles hopes they’ll be able to start doing some of this by next year but cautions that “it’ll take time to build out [ECRI’s] informatics team.”

Despite the many efforts to replace or archive the NGC, however, its shutdown will mean there is no longer a government-run arbiter that’s trusted to serve as an unbiased gatekeeper for and assessor of medical guidelines.

That might not be so bad if AHRQ would relinquish its hold on the NGC’s content and database, so that 20 years of work summarizing and indexing medical guidelines wouldn’t have to be reproduced. Instead, AHRQ has not only defunded and shutdown the NGC, but it continues to prevent others from making the NGC’s content readily accessible again through its silence.