Abstract A large number of small membrane proteins have been discovered in bacteria, but their mechanism of action has remained mostly elusive. Here, we investigate the mechanism of a physiologically important small protein, MgrB, which represses the activity of the sensor kinase PhoQ and is widely distributed among enterobacteria. The PhoQ/PhoP two-component system is a master regulator of the bacterial virulence program and interacts with MgrB to modulate bacterial virulence, fitness, and drug resistance. A combination of crosslinking approaches with functional assays and protein dynamic simulations revealed structural rearrangements due to interactions between MgrB and PhoQ near the membrane/periplasm interface and along the transmembrane helices. These interactions induce the movement of the PhoQ catalytic domain and the repression of its activity. Without MgrB, PhoQ appears to be much less sensitive to antimicrobial peptides, including the commonly used C18G. In the presence of MgrB, C18G promotes MgrB to dissociate from PhoQ, thus activating PhoQ via derepression. Our findings reveal the inhibitory mechanism of the small protein MgrB and uncover its importance in antimicrobial peptide sensing. Significance Statement Small proteins have high prevalence, vast diversity, and primarily regulatory functions in biological processes across all domains of life. However, their mechanisms of action remain largely elusive. In this study, we investigate the mechanism of the small protein, MgrB. It interacts with the sensor kinase PhoQ, rearranges its conformation, represses its kinase activity, and regulates bacterial response to environmental changes. In particular for antimicrobial peptides, MgrB is required for bacteria to have a selective response to this host-exclusive stimulus. Our findings underline the importance of a small protein in bacterial fitness and drug resistance and provide a molecular basis for engineering novel peptide-based regulators.