Diverging opinions about the National Registry of Evidence Based Programs and Practices do not undercut the need for agencies to engage in proper web governance when removing resources.

The homepage of the National Registry of Evidence-based Programs and Practices (NREPP) website as captured by the Internet Archive’s Wayback Machine on July 22, 2018. 

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) decision to retire a major database of evidence-based practice (EBP) materials in 2018 alarmed some researchers, even as they acknowledged its flaws. But experts who spoke to the Web Integrity Project said its replacement leaves much to be desired.

As part of a major overhaul of its web presence that began in October of 2018, SAMHSA removed a broad swath of content from its website and reorganized its content in significant ways. The changes are the subject of our most recent report, which shows that many of the materials removed or deemphasized seemed to embody the “recovery” model of treatment for mental health and substance use disorders. Changes in SAMHSA’s leadership suggest that the agency has been pursuing policies that place less emphasis on this model of care, and the website’s reorganization offers a useful glimpse into the challenges of adapting existing agency messaging under a new policy approach.

When SAMHSA’s site was overhauled, dozens of pages of content were removed, as detailed in our report. But no retired resource has gotten as much attention as SAMHSA’s National Registry of Evidence Based Programs and Practices (NREPP). NREPP, which was taken offline in the summer of 2018, was a large collection of EBP documents designed to provide detailed information about various programs for treating mental illness and substance use disorders. For twenty years, it had served as among the country’s best known clearinghouses of mental health and substance use EBPs, and was widely relied upon to help guide the work of researchers and practitioners.

When NREPP’s retirement became public in early 2018, the news was greeted with suspicion in some quarters, with some experts suggesting the decision was part of a general devaluing of evidence-based medicine under the Trump administration.

“Critics alarmed by the SAMHSA decision said it suggests that what they described as the administration’s skepticism about scientific evidence has now been extended,” as Scientific American summarized the reaction. (As WIP has documented, the administration has also retired another major evidence-based medicine resource, the National Guidelines Clearinghouse, maintained for years by the Agency for Healthcare Research and Quality.) NREPP’s retirement also prompted an inquiry from at least one group of lawmakers.

Following news of its retirement, Elinore McCance-Katz, the then-newly appointed Assistant Secretary for Mental Health and Substance Use at SAMHSA, released a statement that harshly criticized the NREPP database for what she called its “skewed presentation” of the current body of evidence-based treatments in the field. 

Among other criticisms, she condemned what she characterized as an extreme shortage of materials in the NREPP database related to medication-assisted treatment (MAT) for opioid use disorder and EBPs related to schizophrenia. The database returned “few to no results” when searching for related terms, she wrote. Further, because the database was populated by materials submitted by developers — the organizations that create and administer the various treatment programs described — the flaws in NREPP were endemic to its design, McCance-Katz charged. “By definition,” she wrote, programs submitted by developers for review “are not EBPs because they are limited to the work of a single person or group.” In a February, 2018, letter to Senator Maggie Hassan later shared with WIP, McCance-Katz also cited the high cost of some programs in the NREPP database as detracting from its usefulness.

Not everyone viewed NREPP in such stark terms. Dennis Embry, president and senior scientist at the PAXIS Institute, was among those who raised concerns about the database’s demise.

“It’s a huge loss,” Embry told WIP about NREPP’s retirement. The database’s main value, in his view, was  its accessibility for non-specialists, almost like “a Reader’s Digest,” as he described it, for treatment programs that nonprofits or other laypeople could put to use in meaningful ways. Embry noted that a relative under-representation of EBPs related to the subjects singled out by McCance-Katz, which would likely incorporate medical interventions as a central component, wouldn’t be surprising for NREPP. It’s typically not the place for drug related research, he said, which is more likely to fall under the purview of other agencies like the FDA. Even so, Embry said NREPP could have been revamped, and any perceived shortcomings addressed, rather than simply retired. While saying that NREPP was the only national database of its kind, and was indispensable for what it offered a broad range of users, Embry noted that NREPP had “dumbed down” its rating system in recent years, and acknowledged room for improvement.

It was the change in the NREPP vetting process mentioned by Embry, which went into effect in 2015, that attracted the most criticism from other observers, who complained it may have overly simplified the evaluation process. Prior to that time, as explained in a 2018 paper in Substance Abuse Treatment, Prevention, and Policy, NREPP “eschewed overall rankings because their intent was not to tell stakeholders what to do.” Programs evaluated since 2015 had applied a three-tiered ranking system to help readers gauge the appropriateness and effectiveness of the EBPs evaluated.

Some of McCance-Katz’s criticisms echoed a paper published in the International Journal of Drug Policy in 2017, which was highly critical of NREPP, and particularly on those more recently evaluated materials. In that paper, Dennis Gorman, a professor in the Department of Epidemiology & Biostatistics at Texas A&M’s School of Public Health, wrote that the quality of the NREPP database had declined steeply after 2015. By Gorman’s analysis, new materials were being added to the NREPP database at the rate of about eight per month since changes to the review criteria. That pace raised questions, he wrote, about the rigor of SAMHSA’s process. He also argued that 87 percent of these newer materials had “involve[d] some conflict of interest” because of ties between those who had evaluated a particular program and those who developed it.

In a conversation with WIP, Jennifer Sharpe Potter, professor of psychiatry and behavioral sciences at UT Health San Antonio, said that in her view, NREPP had declined in quality in recent years, citing some of the same concerns Gorman identified. 

“NREPP felt like it had gotten dated and a little dusty on the shelf,” Potter told WIP, and she worried that laypeople might not have been able to discern the well-supported, high quality EBPs from some of the lower quality materials listed. The result was that the site was still useful for academics and those equipped to evaluate the quality of materials they found, but less reliable — or even misleading — for the general public.

Shortly after it announced that it would be discontinuing NREPP, SAMHSA launched a new collection of evidence-based practice materials that it calls the EBP Resource Center. But while all of the experts WIP spoke with welcomed the idea of reinvigorating NREPP and strengthening its review process, they all regarded the EBP Resource Center as a lackluster replacement based on their initial review.

“What I would want to know if I was asking SAMHSA what was the intent of this site? Were the objectives the same as the NREPP site, and if not, why?” Potter asked.

There has been an effort to maintain access to the materials once collected on NREPP. As announced this summer, the Pew-MacArthur Results First Initiative has been able to preserve about 80 percent of the content that was once on NREPP, through links captured by the Internet Archive’s Wayback Machine. Sara Dube, the director of the project, told WIP that they made a concerted effort to archive that material after the site went offline, after their own users told them how valuable it had been to their work.

As noted in our report, SAMHSA’s decision not to publicly archive the site risked causing problems for the public. Whatever shortcomings it may have had, simply cutting off access to a widely used database like NREPP is contrary to best practices of web governance as laid out in WIP’s guidelines. 

SAMHSA issued a brief statement to WIP, but did not answer detailed questions about NREPP or the EBP Resource Center. For now, experts told WIP, the new database doesn’t appear to measure up to what NREPP once provided. 

“If you go and download these papers and PDFs, they would not even rate as a good literature review,” Embry told WIP. “They’re polished — nice pictures,” but they only talk about general principles of care, he added.