Abstract In constructing Gene Regulatory Networks (GRNs), it is crucial to consider cellular heterogeneity and differential gene regulatory modules. However, traditional methods have predominantly focused on cellular heterogeneity, approaching the subject from a relatively narrow scope. We present HyperG-VAE, a Bayesian deep generative model that utilizes a hypergraph to model single-cell RNA sequencing (scRNA-seq) data. HyperG-VAE employs a cell encoder with a Structural Equation Model to address cellular heterogeneity and build GRNs, alongside a gene encoder using hypergraph self-attention to identify gene modules. Encoders are synergistically optimized by a decoder, enabling HyperG-VAE to excel in GRN inference, single-cell clustering, and data visualization, evidenced by benchmarks. Additionally, HyperG-VAE effectively reveals gene regulation patterns and shows robustness in varied downstream analyses, demonstrated using B cell development data in bone marrow. The interplay of encoders by the overlapping genes between predicted GRNs and gene modules is further validated by gene set enrichment analysis, underscoring that the gene encoder boosts the GRN inference. HyperG-VAE proves efficient in scRNA-seq data analysis and GRN inference.

Inferring gene regulatory networks by hypergraph variational autoencoder