Why Does the Reproducibility of Search Strategies and Search Results in Systematic Reviews Reduce Over Time? What We Can Do About It?

Farhad
8 min readDec 6, 2023
Reproducibility and Replicability
Reproducibility and Replicability. Source: Turing Way: https://the-turing-way.netlify.app/

There is no excuse; we MUST do our best to report the search methods as reproducible as possible. But what is reproducibility?

In my previous paper titled “Reproducibility and Replicability of Systematic Reviews”, I used the following definition that seems to be still relevant:

Reproducibility is re-conducting the same study, using the same methods and data by a different researcher or team, and replicability is re-doing the same study to gather new data or recollect the data (Patil et al. 2016).

Are Systematic Reviews, Research Studies or Only Literature Reviews?

Some methodologists argue that a systematic review is not a research study but a review, just like any narrative literature review, and we should distinguish review from research. Unlike narrative literature reviews, systematic reviews follow a detailed research question and pre-set protocol involving at least three people. However, one might claim that even narrative reviews can have those properties. What differentiates systematic review from narrative review is Reproducibility.

If a systematic review is reported in a reproducible format so that the reader can follow the methods, repeat the study, and reproduce the same or very similar results, we can call it a research study. Otherwise, we should only call it a narrative review. So reproducibility is even more important for systematic reviews than other research studies. Obviously, all stages of systematic review should be reproducible. I have discussed how using two people or automation at most stages makes the reviews reproducible, but I focus on searching in this post.

Reproducibility in a systematic review search context means the ability to re-run the search strategies in the corresponding sources and reproduce the same or very similar search results in terms of content and number of results.

How do you test if the search…

--

--

Farhad

An Evidence Scientist with a Pinch of Career and Life Lessons