Abstract Genotype-phenotype association is found in many biological systems, such as brain-related diseases and behavioral traits. Despite the recent improvement in the prediction of phenotypes from genotypes, they can be further improved and explainability of these predictions remains challenging, primarily due to complex underlying molecular and cellular mechanisms. Emerging multimodal data enables studying such mechanisms at different scales from genotype to phenotypes involving intermediate phenotypes like gene expression. However, due to the black-box nature of many machine learning techniques, it is challenging to integrate these multi-modalities and interpret the biological insights in prediction, especially when some modality is missing. Biological knowledge has recently been incorporated into machine learning modeling to help understand the reasoning behind the choices made by these models. To this end, we developed DeepGAMI, an interpretable deep learning model to improve genotype-phenotype prediction from multimodal data. DeepGAMI uses prior biological knowledge to define the neural network architecture. Notably, it embeds an auxiliary-learning layer for cross-modal imputation while training the model from multimodal data. Using this pre-trained layer, we can impute latent features of additional modalities and thus enable predicting phenotypes from a single modality only. Finally, the model uses integrated gradient to prioritize multimodal features and links for phenotypes. We applied DeepGAMI to multiple emerging multimodal datasets: (1) population-level genotype and bulk-tissue gene expression data for predicting schizophrenia, (2) population-level genotype and gene expression data for predicting clinical phenotypes in Alzheimer’s Disease, (3) gene expression and electrophysiological data of single neuronal cells in the mouse visual cortex, and (4) cell-type gene expression and genotype data for predicting schizophrenia. We found that DeepGAMI outperforms existing state-of-the-art methods and provides a profound understanding of gene regulatory mechanisms from genotype to phenotype, especially at cellular resolution. DeepGAMI is an open-source tool and is available at https://github.com/daifengwanglab/DeepGAMI .