This paper introduces a deep learning-based approach for calibrating hardware defects in physical-layer key generation (PKG) tasks, focusing on directional-of-arrival (DoA) based key generation in wireless communication systems. The proposed scheme leverages a novel neural network architecture, incorporating residual and self-attention mechanisms, to accurately map spatial features under coherent signals, thereby significantly reducing bit mismatch rates inherent to antenna array imperfections. Through extensive simulation experiments, the method demonstrates improved robustness and effectiveness over traditional calibration techniques and existing deep-learning models, particularly in environments characterized by defect complexity and signal coherence challenges. Our findings offer a promising avenue for enhancing the security of wireless communications by optimizing the performance of PKG solutions.