Spectral analysis using wavelets is widely used for identifying biomarkers in EEG signals. At the same time, Riemannian geometry enabled theoretically grounded machine learning models with high performance for predicting biomedical outcomes from multichannel EEG recordings. However, these methods often rely on handcrafted rules and sequential optimization. In contrast, deep learning (DL) offers end-to-end trainable models that achieve state-of-the-art performance on various prediction tasks but lack interpretability and interoperability with established neuroscience concepts. We introduce GREEN (Gabor Riemann EEGNet), a lightweight neural network that integrates wavelet transforms and Riemannian geometry for processing raw EEG data. Benchmarking on five prediction tasks (age, sex, eyes-closed detection, dementia diagnosis, EEG pathology) across three datasets (TUAB, CAUEEG, TDBRAIN) with over 5000 participants, GREEN outperformed non-deep state-of-the-art models and performed favorably against large DL models on the CAU benchmark using orders of magnitude fewer parameters. Computational experiments showed that GREEN facilitates learning sparse representations without compromising performance. The modularity of GREEN allows for the computation of classical measures of phase synchrony, such as pairwise phase-locking values, which are found to convey information for dementia diagnosis. The learned wavelets can be interpreted as bandpass filters, enhancing explainability. We illustrate this with the Berger effect, demonstrating the modulation of 8-10 Hz power when closing the eyes. By integrating domain knowledge, GREEN achieves a desirable complexity-performance trade-off and learns interpretable EEG representations. The source code is publicly available.