The complex biological mechanisms underlying human brain aging remain incompletely understood, involving multiple body organs and chronic diseases. In this study, we used multimodal magnetic resonance imaging and artificial intelligence to examine the genetic architecture of the brain age gap (BAG) derived from gray matter volume (GM-BAG, N=31,557 European ancestry), white matter microstructure (WM-BAG, N=31,674), and functional connectivity (FC-BAG, N=32,017). We identified sixteen genomic loci that reached genome-wide significance (P-value<5x10-8). A gene-drug-disease network highlighted genes linked to GM-BAG for treating neurodegenerative and neuropsychiatric disorders and WM-BAG genes for cancer therapy. GM-BAG showed the highest heritability enrichment for genetic variants in conserved regions, whereas WM-BAG exhibited the highest heritability enrichment in the 59 untranslated regions; oligodendrocytes and astrocytes, but not neurons, showed significant heritability enrichment in WM and FC-BAG, respectively. Mendelian randomization identified potential causal effects of several exposure variables on brain aging, such as type 2 diabetes on GM-BAG (odds ratio=1.05 [1.01, 1.09], P-value=1.96x10-2) and AD on WM-BAG (odds ratio=1.04 [1.02, 1.05], P-value=7.18x10-5). Overall, our results provide valuable insights into the genetics of human brain aging, with clinical implications for potential lifestyle and therapeutic interventions. All results are publicly available at the MEDICINE knowledge portal: https://labs.loni.usc.edu/medicine.