Photo Credit: Pexels, cc

A 2018 study estimates that every year, over 3 million articles are published in peer-reviewed, English-language journals. Regardless of their backgrounds in research analysis, professionals and policymakers in every field must increasingly make use of these articles and the key data they provide to guide their evidence-based decision making. 

With so many articles, and so much data, it can be hard to determine the accuracy and validity of different work. This quick, three-step guide is designed to help policymakers, advocates, and others interested in data-driven work ensure sources are trustworthy and robust.

Step 1: Look in the Right Places

In the previous paragraph, I referenced that 2018 study because I trust the publisher as an authoritative voice in its field. The publisher, STM (International Association of Scientific, Technical and Medical Publishers), is the leading global trade association for academic and professional publishers in these fields – seems like a good source to turn to for information about academic and scientific research, right? Furthermore, I came across the STM study while reading an article published by a trusted third party source, adding to its credibility. 

But what makes a source trustworthy? In general, look for articles in peer-reviewed publications. Peer-reviewed articles have been reviewed by other experts in the field prior to publication. Basically, the article and its underlying data analysis has received a collective “thumbs-up” from the relevant research community. This isn’t a fool-proof method of ensuring article validity, but it’s a good start. 

Additionally, consider the motives of the study’s author, publisher, and funder. Pay particular attention to  any conflicts of interest. Conflicts of interest do not immediately invalidate a study if they’re clearly disclosed. Make sure the study clearly and transparently states how it arrived at its results and conclusions.

Finally, consider seeking literature reviews, as opposed to individual studies. Literature reviews, sometimes referred to as research reviews, provide an overview of the findings related to a particular topic and  often summarize competing views or interpretations from prominent schools of thought. This can get you up to speed on a particular issue pretty quickly, saving time and effort. 

Step 2: Understand the Methodology 

Any reputable study or research paper should include a section that explains the author’s methodology. If a study does not explain, in detail, how it arrived at its conclusion, you should be highly skeptical of the conclusions. 

Pay particular attention to a study’s sample size, the distinction between causation and correlation, and statistical and substantive significance.

The larger the sample size, the more likely a study’s results will be precise. (The same with greater response rates on a survey). But due to budget and time restrictions, a study may not have a very large sample. If that’s the case, check for an author explanation of the steps they took to compensate for a small number of participants or subjects. 

A common mistake readers often make is confusing “correlation” with “causation”. This is common in news coverage of academic studies, leading to headlines like “New study finds X causes Y!”.  If you read a study that says x is correlated with y, that does NOT MEAN x caused y.

Example 1

If a study claims causation and asserts that x causes y, make sure it accounts for other factors that could be the true underlying cause of y. 

Example 2

Studies that demonstrate both correlation and causation will likely report these findings along with its statistical significance. The statistical significance is an indicator that the results of the study did not just occur by some random chance. It’s important to understand statistical significance, but it’s arguably more important to understand substantive significance. If an effect is substantially significant, that means it is marked or noticeable in the real world. We value statistical significance, but we should also consider substantial significance in our decision-making processes. This is often what lets us know whether a policy change would be really “worth it.” Note, though, that there’s no standard for substantive significance because the magnitude of what’s considered “significant” is different in every field. 

Example 3

Step 3: Determine Applicability

Here at Sunlight, we believe that cities are innovators. We’ve noticed that U.S. cities are constantly looking to each other for the next best policy innovation and are eager to replicate successful programs. It’s important to remember, though, that the findings of a research paper or the success of a certain program may only be valid for the population of that study or program. It’s important to think critically about the specific context of a situation before attempting to replicate a policy intervention. 

Example 4

Something easy to keep an eye out for is publication dates. Something that was valid 20, 10, or even 5 years ago may no longer be valid. It just depends. Look for follow up studies if available, or find out  how each policy initiative has been altered and  implemented in localities similar to yours in size, capacity, governance, and population. At the same time, solid proof of efficacy can and should be accepted. Accept it and move on to your next research question. 

For more information on how to interpret data, we highly recommend seeking out additional resources. Here’s a good place to start.