Abstract In a population-based, cross-sectional study, we assessed age- and sex-specific changes in bone structure by QCT. Over life, the cross-sectional area of the vertebrae and proximal femur increased by ∼15% in both sexes, whereas vBMD at these sites decreased by 39–55% and 34–46%, respectively, with greater decreases in women than in men. Introduction: The changes in bone structure and density with aging that lead to fragility fractures are still unclear. Materials and Methods: In an age- and sex-stratified population sample of 373 women and 323 men (age, 20–97 years), we assessed bone geometry and volumetric BMD (vBMD) by QCT at the lumbar spine, femoral neck, distal radius, and distal tibia. Results: In young adulthood, men had 35–42% larger bone areas than women (p < 0.001), consistent with their larger body size. Bone area increased equally over life in both sexes by ∼15% (p < 0.001) at central sites and by ∼16% and slightly more in men at peripheral sites. Decreases in trabecular vBMD began before midlife and continued throughout life (p < 0.001), whereas cortical vBMD decreases began in midlife. Average decreases in trabecular vBMD were greater in women (−55%) than in men (−46%, p < 0.001) at central sites, but were similar (−24% and −26%, respectively) at peripheral sites. With aging, cortical area decreased slightly, and the cortex was displaced outwardly by periosteal and endocortical bone remodeling. Cortical vBMD decreased over life more in women (∼25%) than in men (∼18%, p < 0.001), consistent with menopausal-induced increases in bone turnover and bone porosity. Conclusions: Age-related changes in bone are complex. Some are beneficial to bone strength, such as periosteal apposition with outward cortical displacement. Others are deleterious, such as increased subendocortical resorption, increased cortical porosity, and, especially, large decreases in trabecular vBMD that may be the most important cause of increased skeletal fragility in the elderly. Our findings further suggest that the greater age-related decreases in trabecular and cortical vBMD and perhaps also their smaller bone size may explain, in large part, why fragility fractures are more common in elderly women than in elderly men.