Ensuring task safety in Human-Robot Interaction (HRI) environments is a critical requirement for reliable and trustworthy robotic systems. AI can be used effectively to estimate robot task safety. However, existing systems suffer from limited data availability and class imbalance, resulting in inaccurate detection of unsafe events. To address these issues, a hybrid speech data augmentation approach is proposed, which combines acoustic and linguistic approaches to train the ML models effectively. The experimentation involves implementing the hybrid augmentation approach, with acoustic transformations for features such as audio level and linguistic transformations for speech data. Results indicate that various Machine Learning models show enhanced performance, achieving up to 0.97 accuracy with the hybrid approach, while the other augmentation approaches achieve lower results, with accuracy ranging from 0.66 to 0.92. In addition, Explainable AI (XAI) strategies are employed to highlight key contributions of significant characteristics such as speech data, audio level, and robot position.