The gesture classification and recognition field offer ample research opportunities due to the growing deaf and hearing-impaired populations and the advancements in vision-based devices. As recognition of hand gestures is crucial to interpreting sign language, gesture recognition systems should consider the maximum number of signs. An ensemble neural network-based robust hand gesture recognition model is presented in this work. Real-time sign detection will be provided by a suggested system in this system. It will facilitate communication with and interaction with challenged people. An American Sign Language (ASL) recognition model can be trained with a convolution neural network (CNN). The proposed model is also suitable for supervised object segmentation. The proposed methodology also performs very well when tested on publicly available datasets of complex hand gestures, with an accuracy rate of 99.111%.