Sensitivity to external demands is essential for adaptation to dynamic environments, but comes at the cost of increased risk of adverse outcomes when facing poor environmental conditions. Here, we apply a novel methodology to perform genome-wide association analysis of mean and variance in nine key brain features (accumbens, amygdala, caudate, hippocampus, pallidum, putamen, thalamus, intracranial volume and cortical thickness), integrating genetic and neuroanatomical data from a large lifespan sample (n=25,575 individuals; 8 to 89 years, mean age 51.9 years). We identify genetic loci associated with phenotypic variability in cortical thickness, thalamus, pallidum, and intracranial volumes. The variance-controlling loci included genes with a documented role in brain and mental health and were not associated with the mean anatomical volumes. This proof-of-principle of the hypothesis of a genetic regulation of brain volume variability contributes to establishing the genetic basis of phenotypic variance (i.e., heritability), allows identifying different degrees of brain robustness across individuals, and opens new research avenues in the search for mechanisms controlling brain and mental health.