Abstract

Interpreting the tumor microenvironment (TME) heterogeneity within solid tumors presents a cornerstone for precise disease diagnosis and prognosis. However, while spatial transcriptomics offers a wealth of data, ranging from gene expression and spatial location to corresponding Hematoxylin and Eosin (HE) images, to explore the TME of various cancers, its high cost and demanding infrastructural needs significantly limit its clinical application, highlighting the need for more accessible alternatives. To bridge this gap, we introduce the Integrated Graph and Image Deep Learning (IGI-DL) model. This innovation, a fusion of Convolutional Neural Networks and Graph Neural Networks, is designed to predict gene spatial expression using HE images. The IGI-DL model outperforms its predecessors in analyzing colorectal cancer (CRC), breast cancer, and cutaneous squamous cell carcinoma (cSCC) by leveraging both pixel intensity and structural features in images. Significantly, across all cancer types, the IGI-DL model enhances the mean correlation of the top five genes by an average of 0.125 in internal and external test sets, rising from 0.306 to 0.431, surpassing existing state-of-the-art (SOTA) models. We further present a novel risk score derived from a super-patch graph, where gene expression predicted by IGI-DL serves as node features. Demonstrating superior prognostic accuracy, this risk score, with a C-index of 0.713 and 0.741 for CRC and breast cancer, supersedes traditional HE-based risk scores. In summary, the approach augments our understanding of the TME from the aspect of histological images, portending a transformation in cancer prognostics and treatment planning and ushering in a new era of personalized and precision oncology.