In recent years, the number of unmanned aerial vehicle (UAV) applications has increased. However, navigating them indoors is still tricky because no GPS signals are available, and the obstacles constantly change. This study exploits the collaboration between convolutional neural networks (CNNs) and reinforcement learning (RL) to overcome these challenges, improving the level of independence of UAVs in indoor settings. We present a new technique known as potential-based reward shaping, which directly incorporates expert knowledge into the reinforcement learning framework. This technique efficiently addresses the problem of sparse rewards that often hinder learning in conventional RL configurations. In addition, we have created a reflective reinforcement learning agent that systematically assesses its actions to prioritize those that result in the most substantial enhancements. We have extensively evaluated the performance of our method with the High-Definition Indoor Navigation (HDIN) dataset to show that our method continuously outperformed current non-introspective RL techniques in terms of decision-making speed and navigation accuracy in this evaluation. The results show that this new CNN-RL combination is feasible and suggests a scalable method for complex indoor navigation problems.