Stimulus-driven attention allows us to react to relevant stimuli (and imminent danger!) outside our current focus of attention. But irrelevant stimuli can disrupt attention as well, for example during listening to speech. The degree to which sound captures attention is called salience, which can be estimated by existing, behaviorally validated, computational models (Huang & Elhilali, 2017). Here we examined whether neurophysiological responses to task-irrelevant sounds indicate the degree of distraction during a sustained listening task and how much this depends on individual hearing thresholds. To this end, we asked N = 47 Danish speaking adults (28/19 female/male, 24 hearing aid users) to listen to continuous speech while one-second-long, task-irrelevant natural sounds (distractors) of varying computed salience were presented at unpredictable times and locations. The task-irrelevant sounds evoked a consistent pupil dilation (PD), neural distractor-tracking (DT) and a drop of neural target-speech-tracking (ΔTT). Within-subject statistical modelling showed that PD and DT are enhanced for sounds with higher computed salience (salience sensitivity), independent of hearing thresholds. Participants with generally larger PD showed stronger ΔTT and performed worse in target speech comprehension. Participants who exhibited a stronger ΔTT for more salient sounds (salience sensitivity) understood less of the target speech. Beyond hearing loss, PD, target speech tracking (and its salience sensitivity) explained 43% of variance in target speech comprehension. We conclude that distraction can be inferred from neurophysiological responses to task-irrelevant stimuli. Our findings represent a step towards neurophysiological assessment of attention dynamics during continuous listening with potential applications in hearing care diagnostics.