Search Filters: What's wrong or right with using them in systematic reviews?

Search filters (Hedges) are search strategies that have somehow been tested or studied to be a good way of finding records relevant to a concept (i.e. an illness, an intervention, a study design, an age group, etc.). The results of such a test or study could have been published or unpublished and could have been rigorous research or an institutional experiment.

We have many search filters, and the most famous ones are the Randomised Controlled Trials search filters recommended by Cochrane.

While many of us love to use such filters and feel like experts when we confidently suggest using them for our research or review, we must be cautious about what we wish for!

1. Nothing is 100%, except for our ignorance

Search filters are not necessarily bringing 100% of relevant records. They also get irrelevant records, and there is no search filter to confidently claim to find all the relevant records in the world of literature. While it is possible to develop a filter and tailor it to a gold standard selection of relevant records to find all of them, in the real world, such a filter may not function well as it will be biased in finding the already known records.

2. Taking a search block out of context

I will write another post about this, but in short, the search filter is only another search block for your search strategy. Taking it out of context could change its performance. Usually, the search filters have been developed based on a test and gold standard record collection from specific journals, published between certain years, for a particular database (i.e. MEDLINE). The performance will change if you change the context (journals, years, or database)!

Many search filters have a published paper in which they usually provide advice on how to use the filter properly and where are the limitations and considerations, without reading those papers it is impossible to get the most benefit from these filters. Even Harry Potter’s invisibility cloak came with an instruction: Use it well!

3. How old is the search filter?

Most search filters are old, and we know that reporting practice in academic literature has improved since the filters were developed. EQUATOR Network's reporting guidelines have become standard for any clinical writer, and journals also mandate the authors to use them. Have the filters been updated? Some of the oldest and probably revolutionary filters I know of are from HiRU at McMaster University. Most of them were developed based on published papers in 2000, and at the time, they were the most exciting search filters. Although we have more updated filters developed in later years, some information professionals stick to nostalgia, nationalism (calling them Canadian or British filters!), simplicity (shorter versus longer filters), or resistance against change. It is not about how we feel or what we like about the filters but about the right things to do.

4. A filter is as good as its gold standard, training, and test collections

As I mentioned in the Context section, to develop a filter, we need several collections, depending on how right we want to do it. The size and diversity of these collections should represent real-world literature, but in most cases, they aren't. Using the records from top journals (BMJ, JAMA, NEJM, and the Lancet) and limiting the number of records for the gold standard, test, or training sets could lead to bias in its performance. The only way to know is to critically read the original paper that publishes the methods used to develop the filter.

5. Known knowns win over known unknowns

A story: In 2009, I ran a search for a health technology assessment project — a confocal scanning microscope for early detection of glaucoma. After everything was finished, the team circulated the final manuscript for comments. Looking at the meta-analyses, I saw a new term that I had never heard of it before. It was a device name, the device that was using confocal scanning technology. You can guess that I had a mini-stroke, picked up my Nokia 3110 phone and called the project manager to say that the search was missing a very important device name that even the clinicians did not know about. I repeated the search and found over 80 new papers that were not mentioning the confocal scanning technology but had the device name. Several of these papers were included in the revised analysis, and the paper was published in 2014 with the original search strategy, but I could finally sleep without guilt. For me, it was a trauma I still remember each time I develop a search strategy.

Another story: I had a similar experience using a search filter developed by NICE information specialists for Health Apps. It generally misses about 50% of papers about mobile apps for the detection of melanoma unless you add the list of known apps to the search. It is not the filter's issue. It is because of reporting and indexing in searchable fields (title, abstract, subject headings). App names, most of the time, appear in full text! If you read the linked paper, you would realise the developers have correctly mentioned all these in their discussion.

When you are searching for a class of drugs or technology, always include the known drug names and devices in the search and do not think that overreliance on search filters, imperfect indexing in databases, or horrible reporting of the title and abstract by the researchers is going to help you. Do not trust any single search method on its own and always use triangulation: ask experts about the most important papers, use Google, Google Scholar, Wikipedia, Emtree, MeSH, read review papers, check the references of included papers, and so on to find all the relevant terms for your search and then test them. If needed, try to add terms rather than removing them from the search filters. Then it’s less like to miss relevant records even though you may get more irrelevant records.

6. Iteration and changes to the search when using a search filter

A recent story: I think you might have heard of EPOC's Low and Middle-Income Countries (LMIC) search filter — or humbly search strategy. Alongside this search strategy, there is a critical appraisal document which is beyond helpful. Last year, I worked on my first evidence gap and map for UNICEF relevant to LMIC. After reading the critical appraisal or evaluation document for the LMIC search strategy, I added several terms, including 'refugee' and 'Palestine', to this search filter. The final results are not out yet, but we found more literature on the refugee population than any LMIC! One would assume that most refugees are from LMIC and go towards High-Income Countries (HIC), and if researchers publish research about them, it may never be found. Can you imagine what we could have missed if I was not reading that evaluation document?

Even if you are using the existing search strategies to avoid re-inventing the wheel, you still need to critically appraise them and modify them if needed.

7. Can we say search filters can be search-resistant concepts like other search blocks?

I already babbled about search-resistant concepts. In short, these are the search blocks or concepts that you better leave them out of the search for reasons explained in the linked post. If you can dump the search filter and deal with the number of results, please do so. I generally add a search filter to handle the huge number of results only when a good search filter exists. Sometimes I don't use the existing search filters and instead, develop a search strategy and test it.

Look at the number of search results with and without the search filter. If the numbers are manageable and you can live without the filter, leave it out.

8. When an effort to develop search filters may fail!

Don't be surprised if you have not seen a search filter for COVID-19 or SARS-CoV-2. Yes, we have databases for COVID-19 literature, but since you still have to search MEDLINE and Embase, you need a good way of searching them. Developing search filters may not be possible when dealing with rapidly growing topics. Instead, having a living search strategy or searching subject-specific databases could do the trick. The main reason filters may not work is that new terms start emerging daily, and your search filter could become outdated as soon as you develop it, and its performance will change. Another reason is that your gold standard, training, and test collection are not representative because the new literature emerges rapidly.

9. Critical appraisal of filters

Yes, you can do it, and it is fun, and maybe you can publish your critical appraisal in your library or personal blog so the others could benefit. Let me know if you have done this, and I promise to add the link here. Critical appraisal of search filters could boost your professional knowledge, leave alone that you can publish it in the Evidence-Based Library and Information Practice journal :D Do it, do it, do it!

10. Search filters that change the databases!

I know you are familiar with the search filters for adverse events and the Embase RCT filter that Cochrane uses to build part of CENTRAL. Embase has shown to be leading in collaboration with librarians. Not only that Embase.com has not embedded some of these search filters into their search features, but MEDLINE and Embase databases have also started correcting their records' index terms based on the findings from Cochrane to make it easier for users to identify the randomised controlled trials. This is how you can change the databases and search practice!

Conclusion

• Search filters are useful tools, and just like any other tools, they will help if you use them after reading the instructions and critical appraisal. Evaluating the filters will help prevent their misuse or abuse.
• Remember that the search filter is just another search block that can be left out of the search if the number of results is manageable.
• You can change the search filters by adding the terms.
• We don't/can't have filters for everything, and even if we do, they may not work well, and occasionally your search strategy may work better than the filters.

If you are excited or confused, look for the next blog posts. I will write about living search strategies, search block libraries, testing the search, de-duplication, screening, and 90 more topics!

If you liked this blog post, please support me by pressing the green Follow button and signing up so I can write more. Email Subscription Dysfunctions. Thank you :D