Evaluation of embodied conversational agents
Research on embodied conversational agent interfaces has produced widely divergent results. We suggest that this is due to
insufficient consideration of key factors that influence the perception and effectiveness of agent-based interfaces. Thus, we
propose a framework for the evaluation of conversational agent interfaces that can systematize the research. The framework
emphasizes features of the agent, characteristics of the user, and
the task the user is performing.
We have conducted experiments within this framework. The first study manipulated the agent's appearance (lifelike versus iconic)
and the nature of the user's task (carrying out procedures versus providing opinions). We found that the perception of the agent
was strongly influenced by the task while features of the agent that we manipulated had little effect. The second study (in progress)
manipulates the initiative of the agent (proactive versus reactive). Initial analysis of the data showed that the participants strongly
preferred proactive agents while initiative of the agent had little effect on their task performance.