Infrared and visible image fusion plays a critical role in combining complementary information gathered from both types of images, thus enhancing the visual quality and the perception in the resulting fused image. Thus, this paper introduces RTAAFusion which is, an innovative image fusion framework that incorporates unique components. This proposed technique employs a Residual Texture-Aware Attention Block Module (RTAABM), meticulously engineered to effectively capture image disparities and texture information. Furthermore, it includes a feature selection attention mechanism that accurately identifies the importance and the weights of the different image features, thereby facilitating a precise and efficient fusion process. The framework also features an Adaptive Decision Block Loss (ADBL), which allows the fusion model to be adjusted to the distinctive characteristics and requirements of various image regions, thus leading to more accurate and targeted fusion results. Comprehensive experiments and comparisons with leading-edge approaches reflected the superior performance of RTAAFusion in terms of visual perception, information conservation, and spatial details across challenging scenarios and a broad range of image features. Therefore, RTAAFusion delivers a fast execution speed and is versatile across different scenarios and image features. This proposed framework shows immense potential for diverse applications within the field of infrared and visible image fusion.