Liver cancer is a common and often fatal disorder that is becoming more commonplace worldwide. An accurate and timely diagnosis is necessary for both effective treatment and patient survival. In machine learning techniques, particularly deep learning, obtaining a large and diverse dataset is still a challenge for deep neural network training, particularly in the medical industry. This paper presents a classification of circulating tumor cells based on the YOLOv8 algorithm. Tumor cell identification and classification can be achieved by utilizing the algorithm’s multi-layer high-level stacking, weight sharing, local connection, and pooling characteristics. The goal is to design a liver cancer classification system that makes it easier and increases the efficiency of doctors in analyzing the results of liver cancer. The models show the absolute the accuracy is 100%, 100%, 98%, 96% to Yolov8n, Yolov8s, Yolov8m, and Yolov8l respectively.

Problem: The frequency of liver cancer is rising worldwide, and it is a common, deadly condition. For successful treatment and patient survival, early and precise diagnosis is essential. The automated classification of liver cancer using medical imaging data has shown potential outcome when employing machine and deep learning (DL) approaches. To train deep neural networks, it is still quite difficult to obtain a large and diverse dataset, especially in the medical field. Aim: This article classifies liver tumors and identifies whether they are malignant, benign tumor, or normal liver. Methods: This study mainly focuses on computed tomography scans from the Radiology Institute in Baghdad Medical City, Iraq, and provides a novel transfer learning (TL) approach for the categorization of liver cancer using medical images. Our findings show that the TL-based model performs better at classifying data, as in our method, high-level characteristics from liver images are extracted using pre-trained convolutional neural networks compared to conventional techniques and DL models that do not use TL. Results: The proposed method using models of TL technology (VGG-16, ResNet-50, and MobileNetV2) successfully achieves high accuracy, sensitivity, and specificity in identifying liver cancer, making it an important tool for radiologists and other healthcare professionals. The experiment results show that the diagnostic accuracy in the VGG-16 model is up to 99%, ResNet-50 model 100%, and 99% total classification accuracy was attained with the MobileNetV2 model. Conclusion: This proves the improvement of models when working on a small dataset. The use of new layers also showed an improvement in the performance of the classifiers, which accelerated the process.