CerealBar

Paper (Suhr et al. 2019) Paper (Kojima et al. 2021)

CerealBar is a two-person collaborative game. We built CerealBar to study natural language understanding in collaborative interactions.

• Two players -- a leader and a follower -- take turns moving around the game board to collect sets of cards and earn points.
• In addition to moving, the leader uses their access to the full environment to plan which set of cards should be collected next, and writes instructions to the follower.
• The follower only has access to a first-person view, so their job is to follow the leader's instructions to the best of their ability. However, the follower can move farther than the leader in each turn.

We crowdsourced interactions between human players in the CerealBar game. We also designed and trained a neural network agent to play as the follower in CerealBar. Our approach makes contributions in modeling, learning, and evaluation. The CerealBar game, data, and modeling approach is described in Suhr et al. 2019 (EMNLP 2019).

CerealBar is an ideal ground to study AI system-human interactions.

Kojima et al. 2021 (TACL 2021) designed a natural language instruction generation system to play as the leader in CerealBar. Kojima et al. demonstrated the system could continually improve by playing with human followers and learning from their behaviors to the system-generated instructions. The source code and data relevant to this project can be found in this Github repository).

This research was supported by grant No. 1750499, a Google Focused Award, an AI2 KSC Award, a Workday Faculty Award, Unity, and an Amazon Cloud Credits Grant. This material is based on work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1650441. We thank Valts Blukis, Jin Sun, and Mark Yatskar for their comments and suggestions, and the workers who participated in our data collection for their contributions.

Also thanks to SQuAD for allowing us to use their code to create this website!

You can check out our YouTube playlist containing the game guide and recorded games between two humans and between a human and the trained agent.

Our Github repository contains the source code and data relevant to CerealBar as well as documentation required to run the game and experiments.

The CerealBar Game is implemented using the Unity game development engine. It can be built as a standalone version (e.g., to replay recorded interactions) or as a web version (e.g., for two players). We are releasing (a) the source code and assets used in the game, and (b) the webapp that deploys the game and supports both connecting two players, and connecting a player to an agent.

We are releasing crowdsourced interactions collected (a) initially for training and evaluation data, and (b) when evaluating our best model in interactions with human players. The data is present on our Github repository, but you can also download it below. We are also releasing utilities scripts to support crowdsourcing on Mechanical Turk.

Download Data (Github)

The Python implementation of our agent is also available on the Github repository. It includes code for simulating the game in Python, data management, the model architecture (in PyTorch), learning, and evaluation.