Abstract Electroencephalography (EEG) and magnetoencephalography (MEG) are used to measure brain activity, noninvasively, and are useful tools for brain research and clinical management of brain disorders. Tremendous effort has been made in solving the inverse source imaging problem from EEG/MEG measurements. This is a challenging ill-posed problem, since the number of measurements is much smaller than the number of possible sources in the brain. Various methods have been developed to estimate underlying brain sources from noninvasive EEG/MEG as this can offer insight about the underlying brain electrical activity with significantly improved spatial resolution. In this work, we propose a novel data-driven Source Imaging Framework using deep learning neural networks (SIFNet), where (1) a simulation pipeline is designed to model realistic brain activation and EEG/MEG signals to train generalizable neural networks, (2) and a residual convolutional neural network is trained using the simulated data, capable of estimating source distributions from EEG/MEG recordings. The performance of our proposed SIFNet approach is evaluated in a series of computer simulations, which indicates the excellent performance of SIFNet outperforming conventional weighted minimum norm algorithms that were tested in this work. The SIFNet is further tested by analyzing interictal EEG data recorded in a clinical setting from a focal epilepsy patient. The results of this clinical data analysis indicate accurate localization of epileptogenic activity as validated by the epileptogenic zone clinically determined in this patient. In sum, the proposed SIFNet approach promises to offer an alternative solution to the EEG/MEG inverse source imaging problem, shows promising signs of being robust against measurement noise, and is easy to implement, therefore, being translatable to clinical practice.