In the last few years, cyberattacks have become more complex, and it is becoming increasingly necessary to establish secure networks. This study examines enhancements to intrusion detection systems (IDSs) with the implementation of machine learning for the categorization of network traffic attacks. For the current study, we utilize four publicly available datasets: CICIDS2017, CIC-DoS2017, CSE-CIC-IDS2018, and CIC-DDoS2019. We examined three machine learning techniques: LightGBM, Random Forest, and XGBoost. Experimental results showed that RandomForest and XGBoost achieved the highest accuracy of 0.99 in both binary and multi-class intrusion detection tasks, maintaining balanced performance with macro F1-scores around 0.86. LightGBM exhibited slightly lower overall performance, but benefited from ANOVA-based feature selection, which improved its recall and model stability. Feature selection also enhanced computational efficiency by reducing feature redundancy while preserving accuracy across models. These results highlight how AI tools could help network security deal with emerging threats and improve the performance of IDS. The study underscores the critical role of feature selection in enhancing model efficiency, hence promoting advancements in automated network security systems that can adapt to evolving cyber threats.

This study provides a detailed comparative analysis of a three-hybrid intrusion detection method aimed at strengthening network security through precise and adaptive threat identification. The proposed framework integrates an Autoencoder-Gaussian Mixture Model (AE-GMM) with two supervised learning techniques, XGBoost and Logistic Regression, combining deep feature extraction with interpretability and stable generalization. Although the downstream classifiers are trained in a supervised manner, the hybrid intrusion detection nature of the framework is preserved through unsupervised representation learning and probabilistic modeling in the AE-GMM stage. Two benchmark datasets were used for evaluation: NSL-KDD, representing traditional network behavior, and UNSW-NB15, reflecting modern and diverse traffic patterns. A consistent preprocessing pipeline was applied, including normalization, feature selection, and dimensionality reduction, to ensure fair comparison and efficient training. The experimental findings show that hybridizing deep learning with gradient-boosted and linear classifiers markedly enhances detection performance and resilience. The AE–GMM-XGBoost model achieved superior outcomes, reaching an F1-score above 0.94 ± 0.0021 and an AUC greater than 0.97 on both datasets, demonstrating high accuracy in distinguishing legitimate and malicious traffic. AE-GMM-Logistic Regression also achieved strong and balanced performance, recording an F1-score exceeding 0.91 ± 0.0020 with stable generalization across test conditions. Conversely, the standalone AE-GMM effectively captured deep latent patterns but exhibited lower recall, indicating limited sensitivity to subtle or emerging attacks. These results collectively confirm that integrating autoencoder-based representation learning with advanced supervised models significantly improves intrusion detection in complex network settings. The proposed framework therefore provides a solid and extensible basis for future research in explainable and federated intrusion detection, supporting the development of adaptive and proactive cybersecurity defenses.