Abstract Many sleep less than recommended without experiencing daytime tiredness. According to prevailing views, short sleep increases risk of lower brain health and cognitive function. Chronic mild sleep deprivation could cause undetected sleep debt, negatively affecting cognitive function and brain health. However, it is possible that some have less sleep need and are more resistant to negative effects of sleep loss. We investigated this question using a combined cross-sectional and longitudinal sample of 47,029 participants (age 20-89 years) with measures of self-reported sleep, including 51,295 MRIs of the brain and cognitive tests. 701 participants who reported to sleep < 6 hours did not experience daytime tiredness or sleep problems. These short sleepers showed significantly larger regional brain volumes than both short sleepers with daytime tiredness and sleep problems (n = 1619) and participants sleeping the recommended 7-8 hours (n = 3754). However, both groups of short sleepers showed slightly lower general cognitive function, 0.16 and 0.19 standard deviations, respectively. Analyses using acelerometer-estimated sleep duration confirmed the findings, and the associations remained after controlling for body mass index, depression symptoms, income and education. The results suggest that some people can cope with less sleep without obvious negative consequences for brain morphometry, in line with a view on sleep need as individualized. Tiredness and sleep problems seem to be more relevant for brain structural differences than sleep duration per se. However, the slightly lower performance on tests of general cognitive function warrants closer examination by experimental designs in natural settings. Significance statement Short habitual sleep is prevalent, with unknown consequences for brain health and cognitive performance. Here we show that daytime tiredness and sleep problems are more important variables for regional brain volumes than sleep duration. However, participants sleeping < 6 hours had slightly lower scores on tests of general cognitive function. This indicates that sleep need is individual, and that sleep duration per se may be a less relevant variable for brain health than daytime tiredness and sleep problems. The association between habitual short sleep and lower scores on tests of general cogntitive function must be further scrutinized in natural settings.