Handwritten ancient documents present unique challenges due to paper aging, ink fading, and blurred handwriting, making their text detection more difficult than standard tasks. Simultaneously, the layout structure is notably intricate, featuring double columns interspersed with single columns along with a blend of images and text, presenting challenges for detection. Therefore, considering the challenges of images from ancient documents, a single stage text detection method on Scale wise Feature Aggregation Module (SFAM) is proposed. It builds on fully convolutional networks to directly generate character level predictions, that identifying redundant and slow intermediate steps Furthermore, by fusing feature maps of different scales to encode information from different receptive field sizes and introducing channel attention mechanisms to allow issues caused by scale variations among different object instances, effective and accurate detection of characters in ancient documents is achieved. In order to assess the effective of the approach, we carried out experiments utilizing the MTHv2 dataset. Our findings indicate that the proposed method surpasses the majority of other text detectors in terms of precision, recall, and F1 score.