The good condition of railway rails is crucial to ensuring the safe operation of the railway network. At present, the rail flaw detectors are widely used in rail flaw detection, they are typically based on the principle of ultrasonic detection. However, the rail detection results analysis process involves huge manual work and the associated labor costs, with low levels of efficiency. In order to improve the efficiency, accuracy of results analysis and also reduce the labor costs, it is necessary to employ classification of ultrasonic flaw detection B-scan image, based on an artificial intelligence algorithm. Inspired by transformer models, with excellent performance in the field of natural language processing (NLP), some deep learning models differ from traditional convolutional neural networks (CNN), gradually emerge in the field of computer image processing. In order to explore the practicality of this model in the field of computer image processing (vision), in the paper, the Vision Transformer (ViT) is employed to train with rail defect B-scan images data and produce a rail defect classification. The model accuracy is more than 90% with the highest accuracy reaching 98.92%.