Abstract How consciousness arises from brain activity has been a topic of intense scientific research for decades. But how does one identify the neural basis of something that is intrinsically personal and subjective? A hallmark approach has been to ask observers to judge stimuli as ‘seen’ (conscious) and ‘unseen’ (unconscious) and use post hoc sorting of neural measurements based these judgments. Unfortunately, cognitive and response biases are known to strongly affect how observers place their criterion for judging stimuli as ‘seen’ vs. ‘unseen’, thereby confounding neural measures of consciousness. Astoundingly however, the effect of conservative and liberal criterion placement on neural measures of unconscious and conscious processing has never been explicitly investigated. Here we use simulations and electrophysiological brain measurements to show that conservative criterion placement has an unintuitive consequence: rather than selectively providing a cautious estimate of conscious processing, it inflates effect sizes in neural measures of both conscious and unconscious processing, while liberal criterion placement does the reverse. After showing this in simulation, we performed decoding analyses on two electroencephalography studies that employ common subjective indicators of conscious awareness, in which we experimentally manipulated the response criterion. The results confirm that the predicted confounding effects of criterion placement on neural measures of unconscious and conscious processing occur in empirical data, while further showing that the most widely used subjective scale, the Perceptual Awareness Scale (PAS), does not guard against criterion confounds. Follow-up simulations explicate how the experimental context determines whether the relative confounding effect of criterion placement is larger in neural measures of either conscious or unconscious processing. We conclude that criterion placement threatens the construct validity of neural measures of conscious and unconscious processing. Significance statement Consciousness has been a topic of scientific research for decades. It is therefore surprising that no consensus has been reached on its neural basis. This may be due to a fundamental problem in consciousness science. Consciousness cannot be observed directly, so that subjective measures are used as behavioral indicators of conscious experience. Here we show that subjective measures cannot reliably reflect experience due to the inescapable arbitrary nature of behavioral criterion placement. We use computational modeling and empirical data to show that the criterion problem has unexpected consequences, such as inflating measures of both conscious and unconscious processing. These findings show that subjective measures are flawed, questioning past work on the neuroscience of consciousness.

Criterion placement threatens the construct validity of neural measures of consciousness