Building Goal-Oriented Dialogue Systems with Situated Visual Context

Sanchit Agarwal, Jan Jezabek, Arijit Biswas, Emre Barut, Bill Gao, Tagyoung Chung

[AAAI-22] Demonstrations

Keywords
Poster Session 4 @ Blue 1, Poster Session 11 @ Blue 1, Poster Session 4, Poster Session 11

Download Paper
Enter the Virtual Venue
Abstract: Goal-oriented dialogue agents can comfortably utilize the conversational context and understand its users' goals. However, in visually driven user experiences, these conversational agents are also required to make sense of the screen context in order to provide a proper interactive experience. In this paper, we propose a novel multimodal conversational framework where the dialogue agent's next action and their arguments are derived jointly conditioned both on the conversational and the visual context. We demonstrate the proposed approach via a prototypical furniture shopping experience for a multimodal virtual assistant.

Sessions where this paper appears

  • Poster Session 4

    Blue 1

  • Poster Session 11

    Blue 1