Macrophages are the most prevalent immune population in colorectal cancer (CRC) and high macrophage infiltration has been associated with both better and worst prognosis. Importantly, according to microenvironment stimuli, macrophages may adopt different polarization profiles, specifically the pro-inflammatory or M1 and the anti-inflammatory or M2, which display distinct functions. Therefore, concomitantly with the number of tumor-associated macrophages (TAMs), their characterization is fundamental to unravel their relevance in cancer. Here, we profiled macrophages in a series of 150 CRC cases by immunohistochemistry, using CD68 as a macrophage lineage marker, CD80 as a marker of pro-inflammatory macrophages, and CD163 as a marker of anti-inflammatory macrophages. Quantifications were performed by computer-assisted analysis in the intratumoral region, tumor invasive front and matched tumor adjacent normal mucosa. Macrophages, specifically the CD163+ ones, were predominantly found at the tumor invasive front, whereas CD80+ macrophages were almost exclusively located in the adjacent normal mucosa, which suggests a predominant anti-inflammatory polarization of TAMs. Stratification according to tumor stage revealed that macrophages, specifically the CD163+ ones, are more prevalent in stage II tumors, whereas CD80+ macrophages are predominant in less invasive T1 tumors. Specifically in stage III tumors, higher CD68 and lower CD80/CD163 ratio associated with decreased overall survival. Importantly, despite the low infiltration of CD80+ cells in colorectal tumors, multivariate logistic regression revealed a protective role of these cells regarding the risk for relapse. Overall, this work highlights how the distinct microenvironments, present at the intratumor, invasive front and adjacent normal mucosa regions, modulate the different macrophage subpopulations, and uncovers their prognostic value, further supporting the relevance of including macrophage profiling in clinical settings.