In large population-based cohort studies, magnetic resonance imaging (MRI) is often used to study the structure and function of the brain. Advanced MRI techniques such as diffusion-tensor (dMRI) or resting-state functional MRI (rs-fMRI) can be used to study connections between distinct brain regions. However, brain connectivity measures are likely affected by biases introduced during MRI data acquisition and/or processing. We identified three sources that may lead to bias, i.e. signal-to-noise ratio (SNR), head motion, and spatial mismatch between MRI-based anatomy and a brain atlas. After quantifying these sources, we determined the associations between the image quality metrics and brain connectivity measures derived from dMRI and rs-fMRI in 5,110 participants of the population-based Maastricht Study. More head motion and low SNR were negatively associated with structural and functional brain connectivity, respectively, and these metrics substantially affected (>10%) associations of brain connectivity with age, sex and body mass index (BMI), whereas associations with diabetes status, educational level, history of cardiovascular disease, and white matter hyperintensities were less or not affected. In addition, age, sex, and BMI were associated with head motion, SNR, and atlas mismatch (all p < 0.001). Based on our results, we strongly advise that, in large population-based cohort neuroimaging studies, statistical analyses on structural and functional brain connectivity should adjust for potentially confounding effects of image quality.