Despite infiltrating immune cells playing an essential role in human disease and the patient response to treatment, the central mechanisms influencing variability in infiltration patterns are unclear. Using bulk RNA-seq data from 53 GTEx tissues, we applied cell-type deconvolution algorithms to evaluate the immune landscape across the healthy human body. We first performed a differential expression analysis of inflamed versus non-inflamed samples to identify essential pathways and regulators of infiltration. Next, we found 21 of 73 infiltration-related phenotypes to be associated with either age or sex (FDR < 0.1). Through our genetic analysis, we discovered 13 infiltration-related phenotypes have genome-wide significant associations (iQTLs) (P < 5.0 x 10-8), with a significant enrichment of tissue-specific expression quantitative trait loci in suggested iQTLs (P < 10-5). We highlight an association between neutrophil content in lung tissue and a variant near the CUX1 transcription factor gene (P = 9.7 x 10-11), which has been previously linked to neutrophil infiltration, inflammatory mechanisms, and the regulation of several immune response genes. Together, our results identify key factors influencing inter-individual variability of specific tissue infiltration patterns, which could provide insights on therapeutic targets for shifting infiltration profiles to a more favorable one.