Abstract Modern single-cell data analysis relies on statistical testing (e.g. differential expression testing) to identify genes or proteins that are up-or down-regulated in relation to cell-types or clinical outcomes. However, existing algorithms for such statistical testing are often limited by technical noise and cellular heterogeneity, which lead to false-positive results. To constrain the analysis to a compact and phenotype-related cell population, differential abundance (DA) testing methods were employed to identify subgroups of cells whose abundance changed significantly in response to disease progression, or experimental perturbation. Despite the effectiveness of DA testing algorithms of identifying critical cell-states, there are no systematic benchmarking or comparative studies to compare their usages in practice. Herein, we performed the first comprehensive benchmarking study to objectively evaluate and compare the benefits and potential downsides of current state-of-the-art DA testing methods. We benchmarked six DA testing methods on several practical tasks, using both synthetic and real single-cell datasets. The task evaluated include, recognizing true DA subpopulations, appropriate handing of batch effects, runtime efficiency, and hyperparameter usability and robustness. Based on various evaluation results, this paper gives dataset-specific suggestions for the usage of DA testing methods.

Benchmarking differential abundance methods for finding condition-specific prototypical cells in multi-sample single-cell datasets