An action evaluation apparatus evaluates a user's action corresponding to an action desired by a system, and determines the ongoing of a story using the evaluation result. The action evaluation apparatus includes a recognizer for recognizing the user's action through a plurality of modals, and an action evaluator for evaluating the user's action based on the recognition result of the respective modals obtained by the recognizer.