Abstract Electrophysiological recordings from freely behaving animals are a widespread and powerful mode of investigation in sleep research. These recordings generate large amounts of data that require sleep stage annotation (polysomnography), in which the data is parcellated according to three vigilance states: awake, rapid eye movement (REM) sleep, and non-REM (NREM) sleep. Manual and computational annotation methods currently ignore intermediate states because the classification features become ambiguous. However, these intermediate states contain important information regarding vigilance state dynamics. Here, we present a new classifier, “Somnotate”, which produces automated annotation accuracies that exceed human expert performance on mouse electrophysiological data, is robust to errors in the training data, compatible with different recording configurations, and maintains high performance during experimental interventions. Somnotate is a probabilistic classifier based on a combination of linear discriminant analysis (LDA) with a hidden Markov model (HMM). A unique feature of Somnotate is that it quantifies and reports the certainty of its annotations, enabling the experimenter to identify ambiguous recording periods in a principled manner. We leverage this feature to identify epochs that exhibit intermediate vigilance states, revealing that many of these cluster around state transitions, whereas others correspond to failed attempts to transition. We show that the success rates of different transitions can be experimentally manipulated and explain previously observed sleep patterns. Somnotate can thus facilitate the study of sleep stage transitions and offers new insight into the mechanisms underlying sleep-wake dynamics. Author summary Typically, the three different vigilance states – awake, REM sleep, and non-REM sleep – exhibit distinct features that are readily recognised in electrophysiological recordings. However, particularly around vigilance state transitions, epochs often exhibit features from more than one state. These intermediate vigilance states pose challenges for existing manual and automated classification methods, and are hence often ignored. Here, we present ‘Somnotate’ - an open-source, highly accurate and robust sleep stage classifier, which supports research into intermediate states and sleep stage dynamics. Somnotate quantifies and reports the certainty of its annotations, enabling the experimenter to identify abnormal epochs in a principled manner. We use this feature to identify intermediate states and to detect unsuccessful attempts to switch between vigilance states. This provides new insights into the mechanisms of vigilance state transitions in mice, and creates new opportunities for future experiments.
