Abstract Quality control (QC) is an important part of all scientific analysis, including neuroscience. With manual curation considered the gold standard, there remains a lack of available tools that make manual neuroimaging QC accessible, fast, and easy. In this article we present Qrater, a containerized web-based python application that enables viewing and rating of previously generated QC images. A group of raters with varying amounts of experience in QC evaluated Qrater in three different tasks: QC of MRI raw acquisition (10,196 images), QC of non-linear registration to a standard template (10,196 images) and QC of skull segmentation (6,968 images). We measured the proportion of failed images, timing and intra- and inter-rater agreement. Raters spent vastly different amounts of time on each image depending on their experience and the task at hand. QC of MRI raw acquisition was the slowest. While an expert rater needed approximately one minute, trained raters spent 2-6 minutes evaluating an image. The fastest was the curation of a skull segmentation image, where expert raters spent on average 3 seconds per image before assigning a rating. Rating agreement also varied depending on the experience of the raters and the task at hand: trained raters’ inter-rater agreement with the expert’s gold standard ranged from fair to substantial in raw acquisition (Cohen’s chance corrected kappa agreement scores up to 0.72) and from fair to excellent in linear registration (kappa scores up to 0.82), while the experts’ inter-rater agreement of the skull segmentation task was excellent (kappa = 0.83). These results demonstrate that Qrater is a useful asset for QC tasks that rely on manual curation of images.

QRATER: a collaborative and centralized imaging quality control web-based application