Traffic noise pollution is a significant source of urban pollution. Roadside noise barriers (RNBs) serve as a primary solution to urban traffic noise pollution, but achieving precise positioning of RNBs requires on-site inspections and manual labeling, which is time-consuming and labor-intensive. Deploying an RNB recognition algorithm on inspection vehicles equipped with cameras and positioning devices is an efficient approach to establishing a high-precision RNB positioning database. We propose a high-accuracy RNB recognition algorithm, RNBFormer. In the model design, we integrate both convolutional and transformer structures to enhance the model's global feature extraction capability while optimizing the representation of local features. Additionally, we incorporate a channel attention mechanism to provide more discriminative features at the channel level. We trained and tested our model on a publicly available dataset, achieving a classification accuracy of 98.93%.