We first propose and study a quantum toy model of black hole dynamics. Themodel is unitary, displays quantum thermalization, and the Hamiltonian couplesevery oscillator with every other, a feature intended to emulate the colorsector physics of large-$\mathcal{N}$ matrix models. Considering out ofequilibrium initial states, we analytically compute the time evolution of everycorrelator of the theory and of the entanglement entropies, allowing a properdiscussion of global thermalization/scrambling of information through theentire system. Microscopic non-locality causes factorization of reduced densitymatrices, and entanglement just depends on the time evolution of occupationdensities. In the second part of the article, we show how the gained intuitionextends to large-$\mathcal{N}$ matrix models, where we provide a gaugeinvariant entanglement entropy for `generalized free fields', again dependingsolely on the quasinormal frequencies. The results challenge the fastscrambling conjecture and point to a natural scenario for the emergence of theso-called brick wall or stretched horizon. Finally, peculiarities of thesemodels in regards to the thermodynamic limit and the information paradox arehighlighted.