Colorectal cancer (CRC) is a multifaceted disease whose development and progression varies depending on tumor location, age of patients, infiltration of immune cells within cancer lesions, and the tumor microenvironment. These pathophysiological characteristics are additionally influenced by sex-related differences. The gut microbiome plays a pivotal role in the initiation and progression of CRC, and shapes anti-tumor immune responses but how the responsiveness of the immune system to the intestinal microbiota may contribute to the sexual dimorphism of CRC is largely unknown. Here, we studied survival, tumor-infiltrating immune cell populations and tumor-associated microbiome of a cohort of n=184 male and female CRC patients and functionally tested the immune system-microbiome interactions in in vivo and in vitro models of the disease. High-dimensional single-cell flow cytometry showed that female patients are enriched by tumor-infiltrating iNKT cells but depleted by cytotoxic T lymphocytes. The enrichment of oral pathobionts and a reduction of beta-glucuronidase activity are distinctive traits characterizing the gut microbiome of women affected by CRC. Functional assays using a collection of human primary iNKT cell lines demonstrated that the gut microbiota of female patients functionally impairs iNKT cell anti-tumor functions interfering with the granzyme-perforin cytotoxic pathway. These results highlight a sex-dependent functional relationship between the gut microbiome, estrogen metabolism, and the decline of cytotoxic T cell responses, contributing to the sexual dimorphism observed in CRC patients with relevant implications for precision medicine and the design of targeted therapeutic approaches addressing sex bias in cancer.