COVID-19-related medical research: a meta-research and critical appraisal

As with many fields, science has been put to the test by the coronavirus pandemic. An international team of researchers, led by French scientists, has demonstrated novel scientific practices around Covid-19. Under the pressure of the number of cases and the urgent need to understand the virus, many research groups have shared results, sometimes to the expense of high scientific standards.

During the first wave of the Covid-19 epidemic, medical research saw a historic number of studies on the coronavirus published in a short period of time. An international consortium of experts in medical methodology undertook the major challenge of evaluating the Covid-19 research during this first wave.

Why should the quality of medical research be assessed?

Since the beginnings of medical research, the quality of scientific results has been subject to evaluation using tools and indicators. The aim is to avoid unreliable results and their unfortunate consequences for both research and medical practice. 

​The recent Lancet Gate case is a regrettable example of results discredited following this evaluation. Shortly after the publication of the study in question showing the potential danger of hydroxychloroquine, the World Health Organization recommended that the use of this treatment be stopped. A few days later, the article was discredited, not affecting the WHO's decision. 

French researchers undErtake the mission to evaluate research on Covid-19

In the context of the ongoing Covid-19 pandemic, researchers in many countries have embarked on a global effort to answer pressing questions around coronavirus diagnosis, treatment and prognosis. The urgency has not given way to a systematic evaluation of the quality of studies during the first wave of the epidemic.

"As far as knowledge of the coronavirus is concerned, we are now in an age of reason." Professor Alexandre Loupy, director of a research team in epidemiology, says, "However, given the huge number of publications related to this virus during the first wave, it was natural for us to form this methodology research group, and to evaluate the quality of medical research. »

In order to carry out this ambitious mission, a consortium was formed by Professor Alexandre Loupy, with French, Chinese and American teams. Together, they designed a large-scale study to evaluate the quality of the research of 10,000 studies on Covid-19 during the first wave.

Among the studies reviewed by experts, only 10% of the articles presented original data. The majority of the studies, about 60%, reflected only opinions and contained no new data, which is unprecedented in scientific history. 
The study also shows a diversity of strategies on the communication of scientific results across countries. China was patient and the researchers waited until they had sufficient data to publish. The United States, England and France, on the other hand, did not wait until they had a large number of cases to start publishing a large number of articles, mainly editorials cautiously anticipating the potential evolution of the epidemic. 

"It was a major piece of work," says Marc Raynaud, methodologist and first author of the study, "Although many studies provided valuable data and analysis, other studies were at risk of disseminating unreliable information. Our findings highlight the need to consider medical studies with caution in the context of a pandemic emergency. »

Lessons to be learned from the scientific treatment of the first wave of Covid-19

The authors of the study emphasise that it is essential to converge towards a balance between scientific rigour and velocity. Indeed, some clinical trials have been published with a limited number of patients and little data. The authors believe that their results should be corroborated by additional data. 

​The emergency has also created misunderstandings that have come under the spotlight. Some studies were retracted after receiving a lot of media attention. Social networks may also have played a role in misinformation by relaying potentially biased studies. 

Important note from the authors of the studY

The consortium would like to point out that the research quality of the clinical trials for the Covid-19 vaccine(s) has not been evaluated. Given the period considered by the study, the results of this work cannot therefore be generalised to these clinical trials, which are known to have a strict design and a robust methodology.

COVID-19-related medical research: a meta-research and critical appraisal

Marc Raynaud, Huanxi Zhang, Kevin Louis, Valentin Goutaudier, Jiali Wang, Quentin Dubourg, Yongcheng Wei, Zeynep Demir, Charlotte Debiais, Olivier Aubert, Yassine Bouatou, Carmen Lefaucheur, Patricia Jabre, Longshan Liu, Changxi Wang, Xavier Jouven, Peter Reese, Jean-Philippe Empana & Alexandre Loupy 
​
BMC Medical Research Methodology volume 21, Article number: 1 (2021) 

Read the paper