What is the minimum number of included studies in a systematic review?
This question has been asked with different wording:
- How many studies do I need for my systematic review? Zero.
- How many studies are enough for a meta-analysis? 2 or more (depends, read on).
- Can I publish a systematic review with no studies? Yes.
These are interesting questions that my students and colleagues ask their supervisors, statisticians, librarians or information specialist. The reason behind the question or the purpose of the systematic review would change the answer from the right and easy answer, which is zero, to more complicated answers such as 2, 5, 10, or as many as possible.
There is no minimum or a maximum number of studies for inclusion or exclusion in a systematic review. Many researchers — by mistake — scream Eureka Archimedes-style when they found nothing for their systematic reviews.
Systematic Review could be done for many purposes and the number of the included study or studies would be dependent on the purpose.
Purpose 1: Coursework, assignment, master's or PhD thesis dissertation
Yeah, you wish! Your supervisor should be an angel to let you get on with no included studies! Even if you have an excellent question that has no answer in the literature, the purpose of conducting a systematic review as part of your education is to learn! To learn a systematic review, you should learn (don't think about doing everything perfectly):
- To manage a team and a project
- To develop a review protocol (proposal)
- To design and run searches with assistance from a librarian
- To select the relevant studies among the search results
- To learn the methods to find the full text of the records
- To critically appraise the quality of the studies using relevant tools
- To pull the data and information out of studies and pool them into a table or figure for data analysis/synthesis
- To write a report
If your systematic review has no included studies, you may learn the first five steps but not the rest of the steps, so your supervisor would insist that you run scoping searches to ensure your systematic review does not end up with zero studies. Depending on your timeline, degree (Undergrads, masters, or PhD), and your question, your supervisor may ask or help you find a systematic review with more than two or more studies.
Purpose 2: Grant Application or Research Proposal or PhD Application
While your literature review would likely have included studies, a lack of studies could possibly make a very strong or very weak case. The strong case is that you have identified an evidence gap and a novel research area. The weak case is that the question you are asking is not answerable! Either it is not asked correctly, or there aren't enough resources to answer that question, and that's why no one has answered it yet.
Evidence Gap: Grant application is a good use of systematic reviews with no included studies to show the evidence gap and inform the practice and policy that hey we have no evidence in this area, and there is a gap to be filled. This is a way to convince funders to put some money aside and let the poor researchers fight over it. Welcome to Academia!
Novelty: Some of the funders or supporters of your research would like to know how 'novel' your research and research question is. In a simplistic view, if there is no other study to answer the question, you meet the criteria for novelty! The novelty has a broader meaning in the research and practice context, and repetition is not necessarily bad for validation, re-validation, and generalisation purposes; however, in many cases, finding no studies means you are answering a question that has never been answered before. In the old days, the funders used to ask for a 'comprehensive literature review', but they finally adapted the new methodologies and now, they ask for a 'systematic review'.
So when applying for a grant, it is acceptable to run a systematic review and find no study that can answer the question with enough certainty. This could be the same when applying for a PhD position, and the funder asks how you will spend/waste their money. They ask you to write a proposal in which you can show the depth of your knowledge by reporting that your systematic/scoping review found a gap that you will fill during your PhD.
Purpose 3: A Publication into Your Empty CV
I understand what an empty CV means, and I know it is tempting to do a literature review than primary research for many with no funding on the horizon. While conducting a systematic review is not necessarily easier or more complicated than any other research study, it is an option if you have between 3–12 months to dedicate.
Some journals will not accept a systematic review without a meta-analysis. Others would ask for at least two included studies. If you have a 'zero-study' systematic review, you should contact and ask the editor-in-chief or the relevant editor before submission.
Who would publish a review with zero included study reviews? You'd be surprised. If you follow the systematic reviewing procedure correctly to answer a real-world question, you can always find a way to publish them. There are many research studies finding "No Evidence". I am sure you can find several published systematic reviews with no included study. Cochrane Database of Systematic Reviews is one of the journals that has published systematic reviews that have not included any studies. Some of these reviews were later updated years later and included new studies. This journal publishes the reviews as long as the reviews follow a rigour methodology. What's important is to highlight that sometimes there are claims, but there is no evidence to support the claims, or there are questions with no answers. Since some of these claims or questions could be crucial, a systematic review with no included study could highlight such importance and gap in the evidence.
While many find it risky to pursue publishing a systematic review with no included study, it is not impossible. Some journals have unwritten subjective criteria in publishing systematic reviews with more than 1 included study. Even a systematic review with a single included study has added value for critically appraising the risk of bias in the evidence.
Purpose 4: Declaring your love for Meta-Analysis and Forest Plot decorated with a Diamond
Some people love to publish a paper with a forest plot; I guess this enthusiasm comes from people's interest in diamonds and the illusion of wealth! Because the forest plot indeed gives a diamond (Find the diamond in the figure below). This is the sort of diamond you may get if you are dating an academic.
Question: How many studies do you need for a meta-analysis? How many effect sizes are needed for a meta-analysis?
Answer A: 2 (technically and acceptably)
2 is the easy and somehow the correct answer where there are no follow-up questions. The definition of Meta-Analysis certainly highlights the statistical "combination" of results of two or more studies. So, theoretically, you need at least 2 studies to run a meta-analysis.
Practically, you need at least 1 study to conduct a meta-analysis and have a beautifully smiling forest plot as long as the software program allows you this option. Just because the software program will enable us to run a meta-analysis it doesn't mean we should do it or the output is useful; however, it is possible:
In addition, the 2 studies are enough if you are working with a Fixed-Effect model because, in this model, a summary based on two studies results in a more precise estimate of the true effect than each study alone. In reality, we would deal more with the Random-Effects model than the Fixed-Effect model.
Answer B: 5 or more (statistical power-wise).
A study reported the power calculation from 1991 meta-analyses from the systematic reviews from the Cochrane Database of Systematic Reviews. It concluded that 5 or more studies are needed to achieve statistical power in a meta-analysis greater than the statistical power in studies that contribute to the analysis (Random-Effects model). We could agree that in most cases, we would be dealing with Random-Effects models (I will write about the models later). Please note that this suggested number is not a rule and may be valid in the context (systematic review of randomised controlled trials). This number can change depending on the usual sample size in the field of study and the assumptions in calculating the power that would require expertise in the review topic as well as statistics.
If interested, read Valentine et al. 2020.
Answer C: Generally, the more, the better
There are other reasons for having more studies. Statistically, a meta-analysis could be done by 2 studies, but what if these two studies are biased or outliers? Including several studies could show the consistency of effect or direction of effect across studies. Another issue is the assessment of "Heterogeneity" across studies and conducting sensitivity analysis. Although looking at the characteristics of the included 2 studies, you can discuss the heterogeneity qualitatively; it is not possible or easy to assess the heterogeneity reliably in a statistical way with 2 studies. The high number of studies would also allow testing publication bias, running meta-regression, and subgroup analysis.
Purpose 5: Declaring your love for funny Funnel Plot
Some people love to have a funny funnel plot to assess the publication bias visually. You can't have a funnel plot to visualise the publication bias with no or one study. It is advised that you should have at least 10 studies. Since many systematic reviews, many never have 10 studies; you can find new love with other statistical methods of discovering the publication bias or do your best to include all studies regardless of their language and publication status.
Conclusion
A systematic review can include 0 or more studies, depending on the serving purpose, the question, and the available resources to answer the review question.
A meta-analysis requires at least 2 studies; however, to reach a statistical power higher than the power of individual included studies, you need to consult with a statistician and an expert in the review field to determine the number of studies or required effect size. Unlike most primary studies that can have a sample size calculated ahead of analysis, the number of included studies in a systematic review cannot be predicted because we don't know the number of available studies that answer the question before running systematic searching, screening, and extracting the data.
If you want to have a funnel plot, you need to have at least 10 included studies.
What makes it a systematic review is a systematic process, not the number of included studies.
While the above questions focus on the ‘number’ and ‘quantity’ of included studies, it forgets about the ‘quality’ of included studies. It does not matter how many studies you include in your review, if they are not of good quality, the quality of your conclusion and evidence will be affected. Remember: Garbage in, garbage out. In most plots above, I intentionally kept the Risk of Bias items to highlight their importance.
If I receive feedback from more experts, I will update this post.
If you liked this blog post, please support me by pressing the green Follow button and signing up so I can write more. Email Subscription Dysfunctions. Thank you :D
