Abstract

Abstract Polarization moments play a crucial role in measuring magnetic fields for nonlinear magneto‐optical rotation (NMOR) atomic magnetometers. However, it is challenging to distinguish between each polarization moment and evaluate its effect on the magnetic resonance response signal in an alkali vapor cell with buffer gas. To address this issue, a method is proposed to identify different polarization moments through the frequency shift of the magnetic resonance response signal. The proportion of each polarization moment is determined, and it is demonstrated that the magnetic resonance response signal is affected by the hexadecapole moment, resulting in a frequency shift and a decrease in signal amplitude. To mitigate this effect, an approach is investigated to manipulate the polarization moments by flipping the phase of the pump light. Ultimately, a 15.19% increase in response amplitude is achieved in the simulated geomagnetic environment within the magnetic shield barrel. The theory and method presented here provide strong support for the study of the polarization moments in an alkali vapor cell with buffer gas, which potentially enhance the performance of NMOR atomic magnetometers.