Abstract

Motivation: Reconstruction challenges for Quantitative Susceptibility Mapping (QSM) offer a common evaluation but are challenging to run and offer only a single snapshot of algorithm performance in time. Goal(s): To develop a QSM challenge with continuous integration (QSM-CI), enabling automatic, transparent evaluation of community-submitted algorithms across diverse datasets and metrics. Approach: QSM-CI was implemented using GitHub Actions and a user interface for displaying metrics and collecting community visual ratings, which inform a qualitative Elo metric alongside quantitative assessments. Results: The QSM-CI prototype implementation is publicly available and has been tested using a range of QSM algorithms. Impact: QSM-CI will facilitate a QSM challenge that allows for continuous evaluations using current and future datasets, algorithms, and metrics. This ensures the continued accessibility of the challenge and continued relevance as new methods, metrics and test data are made available.