The scientific publication score - a new tool for summarizing evidence and data quality criteria of biomedical publications

Dieter Bettin; Thomas Maurer; Ferdinand Schlatt; Simon Bettin

doi:10.5194/jbji-7-269-2022

The scientific publication score - a new tool for summarizing evidence and data quality criteria of biomedical publications

J Bone Jt Infect. 2022 Dec 21;7(6):269-278. doi: 10.5194/jbji-7-269-2022. eCollection 2022.

Authors

Dieter Bettin¹, Thomas Maurer², Ferdinand Schlatt³, Simon Bettin⁴

Affiliations

¹ Department for General Orthopedics and Tumor Orthopedics, University Clinic Münster, 48149 Münster, Germany.
² Orthopedic Clinic Kantonsspital Baselland Liestal, 4410 Liestal, Switzerland.
³ Department of Informatics, Martin Luther University of Halle-Wittenberg, 06120 Halle (Saale), Germany.
⁴ The University Medical Center Hamburg-Eppendorf (UKE), 20246 Hamburg, Germany.

Abstract

The number of biomedical research articles increases by over 2.5 million publications each year, making it difficult to stay up to date. In this study, we introduce a standardized search and evaluation tool to combat this issue. Employing crowdsourcing, a large database of publications is gathered. Using a standardized data entry format, coined the "scientific publication score" (SPS), specific publication results can be easily aggregated, thereby allowing fast and accurate comparisons for clinical questions. The SPS combines two quality dimensions. The first captures the quality of evidence of the study using the evidence criteria defined by the Centre for Evidence-Based Medicine, Oxford, UK. The second is more fine-grained and considers the magnitude of statistical analyses on individual and specific results. From 2014 to 2019, experts of the European Bone and Joint Infection Society (EBJIS) were asked to enter data of relevant publications about prosthetic joint infection. Data and evidence levels of specific results were averaged, summarized and ranked. A total of 366 publications were divided into two groups: (I) risk factors (e.g., host-related factors, pre- and postoperative issues) with 243 publications and (II) diagnostic methods (e.g., laboratory tests, imaging methods) with 123 publications. After ranking, the highest score for risk factors of prosthetic joint infection were calculated by the SPS for anemia (mean $3.50 \pm$ SD 0.91), malignancy (mean $3.17 \pm$ SD 0.29) and previous alloarthroplasty (mean $3.00 \pm$ SD 0.35). A comparison of the full SPS ranking with the ranking determined at the 2018 International Consensus Meeting (ICM) on Musculoskeletal Infection resulted in a Spearman rank correlation coefficient of 0.48 and a $p$ value of 0.0382. The diagnostic methods ranked highest by the SPS were aspirate leucocyte count (mean $3.15 \pm$ SD 1.21), interleukin 6 (mean $3.14 \pm$ SD 1.07) and aspirate (neutrophils over 80 %) (mean $3.12 \pm$ SD 0.63). The comparison to the ICM ranking yielded a Spearman rank correlation coefficient of 0.91 and a $p$ value of 0.0015. Our pilot study evaluated a new tool for the quality assessment of specific results based on the quality of the source publication. The SPS is suitable for a ranking of specific results by evidence and data quality criteria important for systematic reviews.