Autonomous vehicles (AVs) have the potential to revolutionize the transportation system by enhancing road safety, reducing traffic congestion, and freeing drivers from monotonous tasks. Effective exploration is essential for AVs to navigate safely and adapt to dynamic environments. Reinforcement learning (RL) enables AVs to learn optimal behaviors through continuous interaction with their environment. This paper reviews recent RL research on designing exploration strategies for single- and multi-agent AV systems. It categorizes exploration methods based on underlying principles and addresses the challenges. It analyzes key RL algorithms’ strengths, limitations, and empirical performance. By compiling and analyzing the current state of research, this paper aims to facilitate future advancements in AV exploration using RL, offering insights into current trends and future directions in this evolving field.

Highlights: What are the main findings? The paper presents a unified overview of RL-based lane-changing systems by linking maneuver taxonomy, algorithm families, simulators, and evaluation metrics into one structured framework. A comparative analysis shows that no single RL paradigm suits all driving scenarios, revealing distinct trade-offs among value-based, policy-based, actor–critic, model-based, and hybrid methods. What are the implications of the main findings? The findings highlight the need for integrated designs that combine RL with safety layers (e.g., MPC/CBFs), cooperative multi-agent decision-making, and explainable mechanisms to ensure trustworthy deployment. The results motivate the development of standardized evaluation benchmarks and simulation-to-real adaptation strategies to improve robustness and real-world applicability of RL-based lane-changing systems. Autonomous lane-changing is one of the most critical and complex tasks in automated driving. Recent progress in reinforcement learning (RL) has shown strong potential to help autonomous vehicles (AVs) make safe and flexible lane-change decisions in real time under uncertain traffic conditions. In the current studies, there is a lack of a common structure that links RL algorithms, simulation tools, and performance evaluation methods. This paper presents a detailed examination of RL-based lane-changing systems in AVs, tracing their development from early rule-based models to modern learning-based approaches. It introduces a clear classification of lane-changing types—discretionary, mandatory, cooperative, and emergency—and connects each to the most suitable RL methods, including value-based, policy-based, actor–critic, model-based, and hybrid algorithms. Each method is examined for its performance, safety, and computational demands. Furthermore, it reviews major simulation environments, such as SUMO, CARLA, and SMARTS, and summarizes key evaluation measures related to safety, efficiency, comfort, and real-time performance. The comparison shows open research challenges, including model adaptation, safety assurance, and transfer from simulation to real-world driving. Finally, it outlines promising directions for future work, such as cooperative decision-making, safe and explainable RL, and lightweight models for real-time use. This review provides a clear foundation and practical guide for developing reliable and understandable RL-based lane-changing systems for future intelligent transportation.