Autonomous vehicles (AVs) have the potential to revolutionize the transportation system by enhancing road safety, reducing traffic congestion, and freeing drivers from monotonous tasks. Effective exploration is essential for AVs to navigate safely and adapt to dynamic environments. Reinforcement learning (RL) enables AVs to learn optimal behaviors through continuous interaction with their environment. This paper reviews recent RL research on designing exploration strategies for single- and multi-agent AV systems. It categorizes exploration methods based on underlying principles and addresses the challenges. It analyzes key RL algorithms’ strengths, limitations, and empirical performance. By compiling and analyzing the current state of research, this paper aims to facilitate future advancements in AV exploration using RL, offering insights into current trends and future directions in this evolving field.