Summary Human brain structure changes throughout our lives. Altered brain growth or rates of decline are implicated in a vast range of psychiatric, developmental, and neurodegenerative diseases. Here, we identified common genetic variants that affect rates of brain growth or atrophy, in the first genome-wide association meta-analysis of changes in brain morphology across the lifespan. Longitudinal MRI data from 15,640 individuals were used to compute rates of change for 15 brain structures. The most robustly identified genes GPR139, DACH1 and APOE are associated with metabolic processes. We demonstrate global genetic overlap with depression, schizophrenia, cognitive functioning, insomnia, height, body mass index and smoking. Gene-set findings implicate both early brain development and neurodegenerative processes in the rates of brain changes. Identifying variants involved in structural brain changes may help to determine biological pathways underlying optimal and dysfunctional brain development and ageing.