Abstract Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical/production datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 samples across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p-value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’ by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology