Iterative Systematic Reviewing: From a Targeted Search to Labelling and Minimalistic Data Extraction

Systematic Review versus Iterative Systematic Review

It is always recommended to start a systematic review by writing a protocol for a research question. If you are dealing with a broad topic with many research questions and gaps, we recommend scoping reviews, which are again based on preset protocols.

Sometimes, a scoping review also works as a sample size calculation method for future systematic reviews informing the number of included studies for the systematic reviews resulting from the scoping review.

What if writing a protocol for the systematic review is not feasible, or if we have to keep the protocol very flexible?

Tailoring the Systematic Reviewing for Your Needs

It is not always straightforward to choose the type of your literature review from the list of over 50 review types. You may need to tailor the systematic reviewing process to your needs.

We had faced a couple of reviews in the past four years when we dealt with broad systematic reviews without a complete protocol. We had to make pragmatic decisions based on limited available time (3–6 months), a high number of search results (between 100,000–200,000), a limited budget, and a limited number of reviewers.

Example: A cheap intervention that makes the world better, with few manageable side effects. Yeah, do that systematic review, mate!

The Concept: Iterative Systematic Review (ISR)

When dealing with systematic reviews with one or two main concepts in broad topics when it is not possible to have a preset protocol because of a lack of knowledge on the topic or limited resources, the review team writes parts of the protocol as they proceed with tasks. Some broad “What Works” and “Gap Analysis” reviews could be among the reviews that setting a pre-specified protocol may not be possible. Such reviews can follow an Iterative Systematic Reviewing process where they run a more targeted search to get fewer results (less than 20,000). Then, two review team members use double-labelling or double-tagging during the screening to make sense of the evidence. When the screening is done, the team has a better idea of the diversity of the evidence, based on which the team can either narrow down the review into the topics that fit the purpose of the review or continue planning on the presentation of findings. The review team can also use a data extraction form with limited data points (metadata) to save resources. While systematic reviews are known to be conducted based on the planned and pre-specified protocol, it is not always possible. Even the well-planned systematic reviews report deviations from the protocol. Iterative Systematic Reviews can fill the gap for an evidence synthesis when setting a well-planned protocol is not possible, preferred, or desirable.

Characteristics of Iterative Systematic Reviews

1. Too many changes (iterations) to the protocol
2. Protocol development alongside (in parallel with) the systematic review process rather than protocol preceding the systematic review report
3. One broad search block (concepts) or only two search blocks to be combined with AND for search
4. The number of results is higher than 50,000 hits.
5. Limited available time (3–6 months), budget, or staffing
6. Impossible to deliver following a preset protocol in the set time
7. The topic is too broad.
8. The review team is unsure about the type of review (systematic, scoping, rapid review, evidence gap, or evidence map).
9. Protocol registration or publication is possible, but it may not be preferred because of constant changes to the protocol and limited resources.
10. The review will not be updated. There will be a one-off standalone output with no foreseeable systematic review on the horizon.

Solutions when Dealing with Iterative Systematic Reviews

1. Targeted Searching: Applying limitation to the search by searching in Title rather than other textual fields (abstract or subject headings), searching only for English literature, or a limited period (i.e. last 10 years).
2. Priority Screening: screening only the most relevant or potentially relevant records and leaving the records ranked irrelevant by the machine learning model. For more information, see here and here.
3. Labelling During Screening: assigning topics, labels or tags to the relevant records simultaneously as record screening (screening based on titles and abstracts). For more information on Labelling, see here.
4. Minimalistic Approach to Data Charting: rather than a full data extraction, extract only the most important data that is most relevant to fit the review purpose.

Limitations of Iterative Systematic Review

1. The reasons for deviations from the protocol should be reported. This process should only be used when changes to the protocol do not have generalizability implications for the use of resulting evidence in the real world. An example of when we should not use iterations is when pharmaceutical companies run trials of a medication and report outcomes for 6 weeks while severe adverse events appear from the 7th week. If we change the protocol from 3-month data to 6-week data to cover studies reporting 6 weeks of data just because the literature says so (not what the real world or preset clinical question needs), it is a bias, and such a change to the protocol is not acceptable.
2. Iterative Systematic Reviews should only be used when the review cannot fit into one of the existing review types.
3. Acknowledge the possibility of missing studies and limitations of targeted search, priority screening, and minimalistic data extraction.

Conclusions

While I resisted the idea of starting a systematic review without a set protocol for two decades, I’m concluding that we occasionally (1 in 100) need to build the bridge as we cross the river or we cross the bridge when we come to it.

I know many review teams have been secretly using Iterative Systematic Reviewing without reporting it. I tried to de-stigmatise this approach for when you have no other choice.

I tried to organise my thoughts and document this process — like a rapid review — as a new systematic review process rather than a new type of review. This is a newly proposed concept and needs ironing. So, It’s open to discussions.

If you liked this blog post, please support me by pressing the green Follow button and signing up so I can write more. Email Subscription Dysfunctions. Thank you :D