When conducting natural communication in addition to perform verbal communication, a robot partner should also understand non-verbal communication such as facial and gestural information. The word 'understand' for the robot means how to grasp the meaning of the gesture itself. In this paper we propose a smart phone based system, where an emotional model connects the facial and gestural communication of a human and a robot partner. The input of the emotional model is based on face classification and gesture recognition from the human side. Based on the emotional model, the output action such as gestural and facial expressions for the robot is calculated.