Living Evidence Synthesis: On Life, Death, Karma, and Resurrection in Evidence Synthesis
What is the connection? One of the best actresses of the two centuries (Art), one of the best philosophers of the 20th century (Philosophy), and one of the best scientists of the two centuries (Science).
From Sir Karl Popper to Sir Iain Chalmers
Let me start with Downton Abbey’s Lady Violet Crawley, played by Dame Maggie Smith, where she says
My dear, all life is a series of problems, that we must try and solve. First one, and then the next, and the next, until at last we die.
While we always look for solutions, each solution comes with its problems! As Lady Crawley said, and Popper has a book about it, Entire life is problem-solving! It seems that whatever solution we have either creates or reveals more problems, or the solution is itself a problem or a mistake. Some solutions are good when they are first presented, but then they become obsolete and irrelevant. It means we have to correct our mistakes and keep our solutions up-to-date! This is one of the main concepts in Popper’s philosophy, and it is not just about life, dear scientists!
Popper believed that science is the history of corrected mistakes, which is also known as “Self-Correcting Science.”
What we learn at school is already old or may need correction. I mean Medical School! Archie was a doctor in a prisoner-of-war camp with 20,000 prisoners. As the only doctor for many patients with infectious diseases, he only had some aspirin, antacids, and skin antiseptics. Based on his learning from medical school, he expected hundreds to die only from diphtheria. To his surprise, only four died, three of which were because of gunshot wounds. He started questioning what he had learned from medical school and the recommendations. In his book — freely available online — he evaluated medical recommendations in each specialty to find out many of the recommendations were not evidence-based. He writes:
It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials.
Among the specialties Archie evaluated, one of the worst was GO, which stands for Gynecology and Obstetrics. Archie says it could also be “GO ahead with no evaluation.” How do you think a gynecologist named Iain Chalmers—who had similar experiences in Gazza—would feel after reading it?
Chalmers was the one who used Popper’s suggestion and Gene Glass’s meta-analysis to face Archie Cochrane’s criticism welcomingly. Systematic Reviews were started, and some of the systematic reviews that started then are still being updated. That’s also how Cochrane Collaboration started. So, systematically finding, critically appraising, and analyzing the evidence to answer a clinical question and make a medical/clinical/policy recommendation. While the systematic review was a good solution for the problem Archie described, there was another problem. As soon as the search for evidence is finished, the systematic review starts getting old because medical science and technology grow rapidly, and the new evidence can self-correct the old ones. Things change all the time, and the change is the only constant.
Change is all around, and honestly, many people say they hate change, but we get to love it if it means gradually finding solutions for diseases. Change is so common that it is ironically considered the only constant!
Cochrane Collaboration’s solution to change in evidence was to update the systematic reviews every two years to integrate new evidence. Then, they implemented a new criterion for accepting the review submission, which is that the search date for the evidence should not be older than a year. We now know that one size does not fit all, and this 12-month rule may not work all the time. Tell that to SARS-CoV-2! When COVID-19 hit, we needed daily, weekly, and monthly updates; 12 months was too late. In some fields, such as rare or low-burden diseases, we may not have much change in evidence within 12 months, even though we need such evidence; but it does not mean we don’t need constant and continuous surveillance to find evidence. There are some fields in which the change is really constant, and we need to monitor the evidence constantly. Old-style systematic reviews may no longer be enough for all fields.
Why Living Evidence Synthesis (LES)?
For some time, Cochrane Reviews were considered the most up-to-date reviews even though they were planned to be updated every two years. Based on the available resources (technology, funding, volunteers, and time), the two-year updates seemed reasonable at the time.
However, the Evidence Scientists were seeing the gap for topics that needed continuous updates. As mentioned above, in certain fields, the evidence was rapidly growing or changing, and keeping up was not easy; there were too many papers to read and summarise, and clinicians never had time to read and analyze the evidence. In some cases, a single, well-powered study could have challenged the 20 years of small-sized studies and their meta-analysis. At the same time, technology was moving forward in parallel, and it was becoming possible to automate some parts of evidence synthesis to make things faster and more efficient. Even though the idea of Living Systematic Review (LSR) or Living Evidence Synthesis (LES) has been around officially since 2014, COVID-19 gave the last push to more people to finally support LES:
A Living Evidence Synthesis (LES) uses a mechanism to constantly monitor and inform the reviewers about the emergence of new evidence. Such evidence surveillance with some help from automation allows the review to include the new evidence and be updated daily, weekly, monthly, quarterly, or biennually or more than once a year online. The frequency of searches and updates may differ; not every search lead to an update in evidence synthesis.
As LES became possible, shortly after that, the Evidence Scientists started suggesting Living Meta-Analysis, Living Network Meta-Analysis, and, you guessed it right, the Living Clinical Practice Guidelines. WHO is a big fan!
While the Cochrane Database of Systematic Reviews — the first journal with a living model — started publishing in April 1995, it is worth noting that living reviews are not specifically popular in medicine. Living Reviews in Relativity started publishing in 1998, followed by Living Reviews in Solar Physics in 2004, and Living Reviews in Computational Astrophysics in 2015. More and more journals are now trying to adopt a Living Document format and support versioning for the published systematic reviews. Such a living document creates a single source of truth or evidence in answering a research, policy, or practice question.
What’s so special or different about LES compared to systematic reviews?
- Higher frequency of searches
- Relatively higher frequency of updates
- Dependence on automation or semi-automation technologies
- Being online on a live platform using a living document model, supporting and recording citable versioning with unique DOIs
- Usually, reliance on Evidence Maps or Evidence Gap Maps alerts on the availability of evidence for a systematic review of a topic (examples from Infectious Diseases Data Observatory — IDDO).
LES in Neglected Tropical Diseases
War, famine, and pandemics have been the greatest challenges to humankind in history. Consider the role of LES in these three. Since not many readers have seen wars or famine, the latter of the three is no stranger to the world. While Chinese scientists were working and warning about the types of coronavirus, referring to it as a Time Bomb 7 years before the pandemic, and even though we already faced SARS and MERS outbreaks, the world waited and ignored it. Maybe we thought a virus at the depth of the tropical jungle must be harmless until we go to the depth and start poking at it or eating it!
COVID-19, an infectious disease, was not the only target of LES. Many infectious diseases still exist in the status of endemics, syndemics, or pandemics that require constant evidence updates. On the other hand, many Neglected Tropical Diseases (NTD) can benefit from LES for the following reasons:
- Disperse evidence across countries
- Little, low quality, or missing data in evidence
- Higher prevalence in low and middle-income countries with less resources for diagnosis or treatment (distant rural areas and cultural adaptation)
- Reliance on antibiotics and the possibility of antimicrobial resistance
- Studies with generally lower sample size
- Possible syndemics (patients with NTD and HIV)
- Treatment-resistant conditions that require combined, integrated, or complex interventions
- Availability of free and public living maps on such conditions via the Infectious Diseases Data Observatory (IDDO)
We may say what harm a neglected tropical disease can do to us! Wait, déjà vu!
The life, death, karma, and resurrection
As exciting as LES sounds, one must not forget the nature of living is to die. Human beings die from war, famine, pandemics, diseases, accidents/disasters, and old age; LSR and LES also die of famine and old age! If there are no resources to feed them (funding and human resources), they can die young; others may be forgotten. Without an update, they get old and die of old age.
Life after death is not uncommon in nature as there is spring after winter, and there are winters and summers in funding and people’s interest. Some systematic reviews find funding to stay updated; the important LSRs may also eventually find funds to live again. Resurrected LSRs may have different review teams. There might be changes to the protocol and search, but remember that change is the only constant.
What is important to remember is karma; good LSRs with reproducible and sound methodology have a greater chance of survival and resurrecting than bad systematic reviews that hide their methods and data!
May thy LSR be funded, sound, and reproducible.
The past, the present, and the parallel futures of evidence synthesis
The number of LSRs is expected to increase, and the funders will soon realize their values for policy and practice. NIHR in the UK has already announced funding for an evidence synthesis group specific to living evidence synthesis. Australia and Canada have been leading in living guidelines, and the USA is expected to follow. ESRC and Wellcome also announced exciting funding related to LES.
Traditional systematic reviews and many other types of evidence synthesis will continue to survive as LSR is not suitable for all topics; however, they will adopt some of LSR’s characteristics, such as automation, using ML/AI, or evidence surveillance.
It is expected that those automation programs that can provide an ecosystem (from protocol and search to GRADE) all in one place are more likely to become popular, survive, and evolve. In contrast, the other programs will remain task-specific.
We should expect the adaptation of LSR in Social Sciences (soon), humanities, and hopefully politics (very late as usual).
Acknowledgments
I’m grateful to Dr. Charvy Narain, Infectious Diseases Data Observatory, Oxford, UK, for reading and providing constructive feedback on the draft of this blog post.
If you liked this blog post, please support me by pressing the green Follow button and signing up so I can write more. Email Subscription may not function well. Thank you :D
