I am primarily interested in errors within published papers, and what those errors say about the research environment.

University of Sydney

Science Person / Troublemaker

Surgery

Pharmacology

Forensic Metascience

James Heathers's Org

Effect of tissue closure technique on postoperative wound complications

 This is a forensic review of the following paper:https://pubmed.ncbi.nlm.nih.gov/29415592/The study also has a variety of problematic features.These are as follows: (1) Non-obese women included in sample (or not)The paper states:“The age of patients included in this research was between 25 and 35 years, with BMI > 30 (so, we do not have a group of women with BMI below 30 as our focus in the present research was to assess the obese women), ladies who are candidate to perform first CS and have no medical disorders with pregnancy e.g. DM, HTN. All patients were subjected to full history taking and examination. The following patients were excluded women with BMI < 30, who made previous CS, or have any medical disorders with pregnancy as DM, HTN with pregnancy.”The sample is later described in Table 2:BMI: range 23.0–45.0, mean=34.5 ± 3.6, median=34.0One of these statements is incorrect.We cannot reconstruct this distribution well without knowledge of the underlying data. However, a uniform distribution would imply n=121 patients were under the BMI cutoff for inclusion. (2) Registration anomaliesFrom the registration on the NCT website: https://clinicaltrials.gov/study/NCT03354078?tab=history&a=1First Submitted: 2017-11-19First Submitted that Met QC Criteria: 2017-11-21First Posted: 2017-11-27 Study Start: 2017-06-18 [Actual]Primary Completion: 2017-12-23 [Estimated]Study Completion: 2017-12-27 [Estimated]The sample size is given as n=220 [Estimated]. In short, in November 2017, a registration was made for a trial that had already begun in June. It was estimated to be completed a month later, in December. https://clinicaltrials.gov/study/NCT03354078?tab=history&a=2A 2018 update continues:Study Start: 2017-06-18 [Actual]Primary Completion: 2018-08-20 [Estimated]Study Completion: 2018-10-15 [Estimated]The sample size is given as n=220. Then, the collection and study dates were pushed back to August/October 2018. https://clinicaltrials.gov/study/NCT03354078?tab=history&a=3A 2019 update continues:Study Start: 2017-06-18 [Actual]Primary Completion: 2018-08-20 [Actual]Study Completion: 2018-10-15 [Actual]The sample size is given as n=220. Then, the collection and study dates were confirmed completed in August/October 2018. https://clinicaltrials.gov/study/NCT03354078?tab=history&a=4A final 2022 update reverts these dates, specifically:Study Start: 2017-06-18 [Actual]Primary Completion: 2017-11-01 [Actual]Study Completion: 2017-11-23 [Actual]The sample size is given as n=400. In other words, according to the registration, the study was confirmed as (1) completed sooner, backdating the study completion to November 2017, contradicting Versions 2 and 3, and (2) retrieves n=180 additional participants, even though it was previously listed completed at n=220, contradicting Versions 1, 2, and 3, and (3) goes from partially registered in advance (Version 1 lists the study as in progress), to registered fully after completion.Finally, the paper itself asserts: “Four hundred pregnant women were recruited for this study who accepted to share in the present study in the period from June 2017 to December 2017” which reverts to the original Version 1 registration.This is extremely incoherent. (3) Improbable ResultsOne feature we have seen previously in untrustworthy papers is that extremely significant results hide behind significance barriers. An example would be: “A is bigger than B, p<0.01” where p=0.000005. In many scientific fields, this would be an unlikely result, but the true p-value is only available on recalculation and escapes detection in the published paper.Hence, I recalculated several results listed as “p<0.001”.BMI: t = 6.5624, p=1.677e-10. This is a Cohen's d ~=1!Estimated blood loss: t = 4.0572, p=0.00005987.Subcutaneous tissue depth: t=3.6770, p=0.0002686.Suture type: The chi-square statistic is 15.4724. The p-value is .000084. (4) Additional ConcernsAfter writing the above, I found the following paper, with the abstract thus:Concerns about Data Integrity of 22 Randomized Controlled Trials in Women's HealthAbstractObjective During a review on postpartum hemorrhage, we identified randomized controlled trials (RCTs) of one author conducted at the same time and place for the same condition, with large differences in baseline characteristics. We assessed the data integrity of the RCTs of this author.Study Design We undertook a focused analysis of the data integrity of all RCTs published by Dr. Ahmed M. Maged. We examined the studies for clinical logic and made pairwise comparisons of baseline characteristics and outcomes between trials. We used mathematical methods to assess whether the distribution of baseline characteristics was compatible with chance.Results Between March 2015 and December 2019, Dr. Maged published 22 RCTs (n = 3,722). The median number of participants randomized per center per month was 32 (range = 1–89). Fifteen studies were either not or retrospectively registered, with one study registered 1 year after publication. One study was submitted for publication prior to the completion of the described study period. There were many unusual findings in the studies, including biologically implausible occurrences such as the absence of an association between gestational age and birthweight in seven studies and very different body mass index between three trials, which ran at the same time in the same hospital on the same topic as well as unlikely occurrences such as limited participant drop outs. One paper contained considerable text duplication and identical data to that in a paper published by a different author group from a different hospital, with both papers submitted at the same time. Mathematical analysis of the baseline characteristics of all 22 trials indicated that at least some of the reported baseline characteristics were unlikely to be the result of proper randomization.Conclusion Our analyses of the 22 RCTs of Dr. Maged suggest potential data integrity issues in at least some of them. We suggest that journals investigate according to the Committee on Publication Ethics guidelines. The procedures demonstrated in this paper may help to assess data integrity in future attempts to verify the authenticity of published RCTs.Key PointsWe identified a number of findings biologically implausible in RCTs by Maged.Monte Carlo simulation found pooled data of Maged RCTs were unlikely result of proper randomization.Textual overlap and almost identical data were found between a Maged paper and another paper.The methods we described may be useful for future efforts in validating scientific data integrity.This paper describes a strong overlap in time/sample between this paper and another author who produced the work discussed in the abstract above. There may be some overlap between my findings and this work, but the paper is inaccessible. (5) Further Details400 women were recruited, 3 lost to follow-up. While this is unlikely, it is technically possible.A later retrospective study with a much larger sample sized addressed the same question as this paper and found results which were essentially opposite A response to this study criticized it heavily for being under-specified. While I think these criticisms are entirely valid, they do not address the central concern of the data being untrustworthy. (6) SummaryI have no confidence in the accuracy of this study.   