The aorta is the largest blood vessel in the body, and enlargement or aneurysm of the aorta can predispose to dissection, an important cause of sudden death. While rare syndromes have been identified that predispose to aortic aneurysm, the common genetic basis for the size of the aorta remains largely unknown. By leveraging a deep learning architecture that was originally developed to recognize natural images, we trained a model to evaluate the dimensions of the ascending and descending thoracic aorta in cardiac magnetic resonance imaging. After manual annotation of just 116 samples, we applied this model to 3,840,140 images from the UK Biobank. We then conducted a genome-wide association study in 33,420 individuals, revealing 68 loci associated with ascending and 35 with descending thoracic aortic diameter, of which 10 loci overlapped. Integration of common variation with transcriptome-wide analyses, rare-variant burden tests, and single nucleus RNA sequencing prioritized SVIL , a gene highly expressed in vascular smooth muscle, that was significantly associated with the diameter of the ascending and descending aorta. A polygenic score for ascending aortic diameter was associated with a diagnosis of thoracic aortic aneurysm in the remaining 391,251 UK Biobank participants who did not undergo imaging (HR = 1.44 per standard deviation; P = 3.7·10 −12 ). Defining the genetic basis of the diameter of the aorta may enable the identification of asymptomatic individuals at risk for aneurysm or dissection and facilitate the prioritization of potential therapeutic targets for the prevention or treatment of aortic aneurysm. Finally, our results illustrate the potential for rapidly defining novel quantitative traits derived from a deep learning model, an approach that can be more broadly applied to biomedical imaging data.