Need Help with Including Non-English Language (NEL) Studies in Systematic Reviews?

Medical Translation Services for Systematic Reviews

Note: In the Systematic Review context, Americans use Data Abstraction instead of Data Extraction. In Scoping Review context, it may be called Data Charting; however, Social and Health Scientists sometimes use Data Extraction and Data Charting interchangeably.

I’m not going to discuss why it is important to include Non-English Language (NEL) studies language, but if you are interested here you can find a summary. In this post, I will discuss the factors to consider when deciding to include and deal with NEL studies. The big mistake is that the systematic reviewers forget to include how they would deal with NEL studies in their protocol. I rarely see a systematic review protocol mentioning a proper protocol for dealing with NEL. They usually mention whether they will include NEL or not. They don’t even have a budget or consider it in their Gantt chart time-wise. The trouble starts later!

Rasmussen et al. 2018 reported that “the most frequently mentioned challenge to including non-English studies was a lack of resources (funding and time) followed by a lack of language resources (e.g. professional translators).

There are a lot of discussions on how to deal with Non-English Language (NEL) studies in systematic reviews. The short answer is: With No Discrimination!

I hope the answer saves you time not to read the rest of this post, but if you are still curious, please read on :)

Routine Data Extraction Process in Systematic Reviews

Systematic reviews require at least a team of three researchers or reviewers because the following stages of the systematic reviews require between 2–3 people depending on the systematic review’s protocol:

1. Screening the records
2. Screening the reports
3. Screening the studies
4. Extracting data
5. Entering data

If we need two reviewers to screen the English records, reports, and studies and to double-check data extraction and data entry, we ideally would need those two to screen the NEL ones. The only problem is the language barrier, and translation seems to be the best solution.

Step 1: Choose Human/Machine Approaches for Medical Translation in Systematic Reviews

1. Two NEL Speakers for Eligibility Check. Asking two NEL speakers only to extract the PICOS information to check the eligibility before wasting time on data extraction in case it’s excluded. This is an acceptable approach, as it happens the same way for English studies.
2. One NEL Speaker for Data Extraction. It’s unacceptable as this is not the practice for English studies. Usually, one reviewer extracts and the 2nd one checks. They resolve the differences among themselves or ask a 3rd reviewer for help in resolving the differences.
3. One NEL Speaker for Data Extraction and Two English Speakers for Routine. So the two reviewers can follow the routine practice as they do for English studies. This approach is partially acceptable only if the team has no or limited resources because if the translator makes a mistake, no one will know.
4. 1st Human NEL Speaker or Translation Programme for Full Translation and 2nd Human NEL Speaker to Double-Check, and Two English Speakers for Routine. This is the acceptable approach as it reduces the error.

Step 2: Choose Between Full Translation versus Partial Translation

Another question is whether we should translate the paper fully or only the most important part. We found the full translation better for several reasons:

1. NEL Abstract: Sometimes, the NEL abstract contains data different from the English abstract. These data are also good from cross-checking the tables and results section to find discrepancies.
2. Introduction: The introduction of a paper contains background information that can be used in systematic reviews and sometimes highlights the problems, history, or statistics that the review team did not know.
3. Discussion: The discussion usually cites relevant studies comparable to the study in hand, and such literature could possibly be included in the review.
4. Discussion: This section also included the limitations and strengths of the study (usually in the last paragraph and before the Conclusion), which are very helpful in the quality appraisal or assessment of the risk of bias.
5. Other Information: references, footnotes, appendices, funding information, or contact information (email) could all contain information useful for the sake of the systematic review.

If you have limited resources (time, team, money), then partial translation might work as well for you, but you can add a limitations section in your systematic review to say you had a partial translation.

Step 3: Using Translation Software Programmes for Medical Translation and Knowing Their Limitations

We used a machine for the first level of translation where possible. Here are the things to consider:

1. The Original Language: Existing programmes use machine learning, but are not good with all languages, so human translators are better for certain languages. Our experience shows that Machine works well with Latin-based languages and struggles with Arabic, Asian, Farsi (Persian), Hebrew, and Russian.
2. Quality of Document: The machine’s success is dependent on the quality of the uploaded document. If your document is in PDF and if PDF has low quality, you better find a better PDF or prepare for a lot of editing.
3. Level of Service: The existing programmes deliver different levels of services, from free to premium. Usually, the more you pay, the better the translation. If you use the free version, prepare the human translator for more work. Usually, the programmes use different machine learning models to deliver better quality.
4. Format of the Document: MS Word is probably the best format we have worked with. The main reason is that in some PDFs and scanned reports, it is not easy for the machine to detect the end of lines and paragraphs, so the structure of the tables or the labels and numbers in the figure. In DOC or DOCX format, it is easier for the programmes to identify and process the text, table, and figure structure. You may need another programme to convert PDF files into DOC and DOCX. You will also need time to edit the output as some characters might not be recognised during the OCR process (Optical Character Recognition):

During OCR, the scanned file, which is a picture file (usually .jpg, .jpeg, .gif, .png, .tif, .tiff, or a Scanned PDF) where you cannot search, edit, select or copy text, will be converted to a text file (usually in .txt, .DOC, .DOCX, .ODF, or a searchable PDF) where you cannot search, edit, select or copy text. This process may not be perfect and some editing of the output may be required. Modern scanners allow users to choose the scna output to be a unsearchable file or a searchable file.

Step 4: Free Volunteers or Paid Translators

Of course, the majority of the systematic reviewers would think of conducting their review as cheaply as possible but also as fast as possible, and the cost and time are sometimes causing conflicts:

A. Volunteers

Since I have voluntarily extracted data for over 60 reviews and 19 Cochrane Review Groups, I have some experience to share.

1. No Deadlines: Volunteers are free but do not have to deliver on time or with high quality. They are doing a favour and receive nothing but an acknowledgement. If they are not motivated by serving humanity or the topic of the review, you should expect delays.
2. Priorities Change: Volunteers may suddenly get busy with more important tasks and decide to decline their help. Many would not prioritise a mention in the acknowledgement over their paid to-do list.
3. Volunteer Management: Volunteers are free labour and are not easy to find. Finding, managing and keeping them could take resources, especially if you are dealing with several NEL studies in several languages.
4. Abusive Practices: Some volunteers may feel obligated or pushed to help with translation, which may lead to unethical and abusive practices. An example is when a professor, a supervisor, a line manager, or a university lecturer asks their students or staff to help with translation. It is not easy to decline, particularly if the way it’s been asked is intimidating.
5. Authorship Protocol: There is no protocol on how much data extraction qualifies one for authorship. In some reviews, a translator extracts data for over 60% of studies and works more than others without being considered an author. Again, this may cause unethical practices.
6. Seed Fund: A positive lesson is that I started my first work with Cochrane as a volunteer about 12 years before officially being employed by a Cochrane group. When extracting data from NEL studies, I never thought I might end up working for this organisation for 10 years, but 12 years from now! So volunteering in fields related to your interests is planting a seed that may or may not grow in the next 2–100 years.

Volunteering is the investment of your time for a good cause, and it may pay off in the short-term or the long-term either for you or those you’ve helped them.

B. Paid Services

1. Costs: The best services have qualifications and certifications that allow them to deliver the best quality translation services. As a result, their costs will be high; however, you will have some quality guarantee.
2. Expertise in Translation: such services are usually experts in translation rather than systematic reviews. As a result, they may translate a study that you may exclude.
3. Expertise in Systematic Reviews: translation companies do not have expertise in systematic reviews. A systematic reviewer would usually ask for PICOS or your protocol for an eligibility check. Translation services don’t provide such a service. Additionally, a systematic reviewer would provide notes or observations that a translator won’t—for example, noticing a mistake in using SE instead of SD or reporting numbers in tables. Systematic reviewers’ critical thinking skills and background information could help assess the risk of bias, for example, a change in the healthcare system that might have affected the intervention or discriminating ethnical minorities from participants.
4. Individuals vs Companies: Companies give more security in terms of reliability and stability. Individuals would be cheaper for not having admin costs and VAT, but they can disappear if you don’t know them closely.
5. Supporting LMIC: Some companies use freelancers from Low and Middle-Income countries, and any income for such individuals could help their cost of living.
6. Protocol for Acknowledgement: Since you pay, you should not be worried about technicalities for authorship or acknowledgement. The companies usually have a set protocol for being acknowledged in the final report.

What we did for this project

In early COVID-19 lockdown days, a librarian posted to a library mailing list that a researcher needed help with the translation of 17 studies from 7 languages (Chinese, Danish, French, German, Persian, Portuguese and Russian language) into English. Systematic Review Consultants LTD contacted the researcher to help. The UN commissioned their research team to conduct a meta-analysis. Among those who approached this research team, they requested the costs and agreed to work with us. We used machine translation to save time and sent the translated reports to two fluent speakers to check and revise. An English speaker edited the final reports before sending them to the research team. At least 7 comments/questions were raised from the translations, and we requested the NEL speakers to answer them. Three out of these 7 were translation errors detected by the research team!

Even though we delivered the task within two weeks, I wanted to share this story with you to show that even after being translated by machine and checked by two NEL speakers and an English editor, there were still things that needed to be corrected. There are lessons in this experience for those who wish to learn.

Conclusion: Automation Helps, Semi-Automation Rocks!

The paper resulting from this project is now published, we’ve been acknowledged, and I received permission to write about our experience. We learned a lot from this experience, and we have been given more translation tasks from different teams working on other WHO projects; however, we are sharing our experience to allow others to find an efficient way to deal with NEL studies in their systematic reviews:

1. Write how you will deal with NEL in your protocol and before starting your systematic review. It is a time-consuming task to handle, and you need to have plan B in case your plan A does not work.
2. If you are applying for funding, add costs for translation, ordering full-texts, information specialist services, and statistical support.
3. Machines help, but they are still in progress, and their quality depends on the abovementioned factors. The user’s discretion is advised.
4. BIG ONE: Deal with NEL studies exactly how you deal with English studies. Once you have the English translation of reports, follow the standard protocol for data extraction. No discrimination!
5. Professional translation services cost more for a good reason. Even those services might be challenging. Leave room for error all the time.

Share your views, and I will update this post :D

If you liked this blog post, please support me by pressing the green Follow button and signing up so I can write more. Email Subscription Dysfunctions. Thank you :D