The exact mechanisms through which the APOE gene influences Alzheimers disease risk (AD) are under intense investigation. Yet, much remains unknown about how APOE alleles interact with age, sex, and diet to confer vulnerability to select brain networks. To address this question, we used mouse models that carry the three major human APOE alleles, and have a mouse like, or humanized innate immune system. This was realized through the replacement of the mouse with the human inducible nitric oxide synthase (iNOS) gene, leading to a lower immune response. Our study combines advanced, accelerated diffusion imaging methods with novel statistical analyses to reveal global and local brain graph differences, associated with each of the following risk factors for abnormal aging and AD: age, APOE genotype, sex, diet, innate immune response. We used a sparse logistic regression model (GraphClass) with complimentary pairs stability selection to identify small and robust subnetworks predictive of individual risk factors. The comparison of APOE3 versus APOE4 carriage identified 773 edges, with no more than 40 false selections; age resulted in 300 high selection probability edges with <5 false selections, sex resulted in 540 high selection probability edges with <7 false selections; diet in 1626 high selection probability edges with <31 false selections; and humanizing NOS in 411 edges with <9 false selections. Our results revealed widespread network differences due to age and diet, including the temporal association cortex, and also regions involved in response to threatening stimuli, such as the amygdala and periaqueductal gray. Sex associated differences affected regions such as the thalamus, insula and hypothalamus, but also fimbria and septum, involved in memory processes. APOE genotype was associated with differences in the connectivity of memory related areas, and also in sensory and motor areas; while diet and innate immunity (HN) were associated with differences in insula and hypothalamus connectivity. We pooled these models to identify common networks across multiple traits, giving insight into shared vulnerability amongst the risk factors. We identified 63 edges out of the total 54,946 edges (0.11% of the connectome) as common to all risk factors tested. Our results revealed common subnetworks vulnerable to several risk factors for AD, in an approach that can provide new biomarkers, and targets for therapies.