Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria | Summary

This article explains the key points of Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria.

• 2022-01-05 (arXiv), Cooperative AI Workshop, 35th Conference on Neural Information Processing Systems (NeurIPS 2021)
• Kopparapu, Kavya, Duéñez-Guzmán, Edgar A., Matyas, Jayd, Vezhnevets, Alexander Sasha, Agapiou, John P., McKee, Kevin R., Everett, Richard, Marecki, Janusz, Leibo, Joel Z., Graepel, Thore.
• Harvard University, DeepMind
• Paper

Read this article in Korean

Summary

• The paper studies how agents choose whom to cooperate with when other players’ team memberships and motives are hidden. In a two-team social deduction game, agents trained with hidden-role information produced three distinct patterns of play across 10 runs, while all 10 runs of the standard baseline were dominated by Impostor wins through freezing.
• Hidden Agenda places four Crewmates, who collect fuel, against one informed Impostor, who can freeze them. Agents learn through reinforcement learning, in which actions are shaped by rewards, and can coordinate through repeated public votes without natural language.
• The Hidden-Agenda Agent uses hidden roles and player distances during training and learns to predict roles from observations; these privileged inputs are absent from its action policy at evaluation. The reported outcomes are patterns in the learning dynamics of fixed groups, not proven game-theoretic equilibria.

1 Introduction

Social deduction games require players to infer hidden roles from behavior while cooperating with allies and competing against opponents. Hidden Agenda extends this setting into a 2D world where movement, tasks, freezing, and public votes provide evidence about players’ intentions.

2 Hidden Agenda Environment

Hidden Agenda is implemented in Lab2D using the Melting Pot component system [4, 20]. In every experiment, four Crewmates do not know other players’ roles, whereas one Impostor knows all roles; avatar colors are randomized independently of role.

The corner fuel rooms and central deposit point make Crewmates travel through shared space, creating opportunities for the Impostor to intercept them and for other players to observe behavior.

2.1 Overview of Gameplay

During the situation phase, players collect and deposit fuel, carrying at most two cells, while the Impostor can permanently freeze Crewmates. A freeze action witnessed by an active Crewmate outside its radius, or 200 timesteps since the previous voting phase or episode start, triggers a 25-timestep voting phase; a player receiving at least half of active players’ final votes is jailed.

The diagram links physical actions to a public voting phase and shows the return to movement after 25 timesteps.

Crewmates win by depositing enough fuel or voting out the Impostor; the Impostor wins when all but one Crewmate are frozen or voted out. An episode is a draw after 3000 timesteps.

2.2 Environment Details

Each agent observes an 88 × 88 × 3 red-green-blue (RGB) image of its local view, its inventory fraction, global fuel progress, and a 5 × 7 vote matrix. Movement and turning are available in the situation phase, while voting or abstaining are available during voting; inactive players cannot act.

Winning and losing give each team member +4 and −4 respectively. Shaping rewards, intermediate learning signals, give Crewmates +0.25 each for collecting and depositing fuel, the Impostor +1 for a successful freeze, and a frozen Crewmate −1.

3 Reinforcement learning model

The baseline is asynchronous advantage actor-critic (A3C), a reinforcement learning architecture with policy and value predictions [23]. It combines a two-layer convolutional network with 16 and 32 channels, a two-layer multilayer perceptron with 64 units per layer, and a long short-term memory network (LSTM), which maintains a recurrent state, with 128 hidden units and an unroll length of 100.

Training uses V-Trace importance sampling to adjust experience [11] and a contrastive predictive coding loss that distinguishes nearby timepoints through LSTM states [25]. RMSProp uses a learning rate of 0.0004, decay of 0.99, no momentum, and minibatches of 16.

3.1 Hidden-Agenda Agent

The Hidden-Agenda Agent conditions its training critic, which estimates value, on five-element vectors of player roles and distances. An auxiliary cross-entropy loss with coefficient 1.0 trains a 32- and 16-unit predictor to infer roles from the LSTM state; neither privileged roles nor distances are available to the actor at evaluation.

The shaded actor processes observations to choose actions, while player identity and distance support the value estimate and role-prediction training outside the evaluation actor.

3.2 Co-training and evaluation of agents

Five agents co-train in a fixed group: one always plays the Impostor and four always play Crewmates, with colors randomized each episode. Parameter snapshots are evaluated during training using episode returns, win-condition frequencies, and the role representation loss.

4 Experiments

The experiments first search game parameters and then compare standard A3C with the Hidden-Agenda Agent under the same co-training protocol.

4.1 Game parameter sweep

The non-exhaustive sweep used one random seed per setting because of resource constraints. The selected settings were 32 fuel cells to win, voting every 200 timesteps, a 50-timestep freeze cooldown, +0.25 fuel collection and deposit rewards, ±1 freeze rewards, and no additional voting reward.

4.2 Experimental setup

Each architecture was trained for 1 billion timesteps in each of 10 runs with different random seeds. The baseline measures strategic behavior without the Hidden-Agenda Agent’s two modifications.

4.3 Co-training of agents converges to three distinct equilibria

Among Hidden-Agenda Agent runs, Crewmate wins by vote dominated 3/10, Crewmate wins by task 2/10, and Impostor wins by freezing 5/10. All 10 baseline runs converged to a freeze-dominated outcome, although voting accounted for a larger share of outcomes than in the corresponding Hidden-Agenda pattern.

The examples distinguish common task and avoidance behaviors from partnering, voting, and isolating behaviors associated with particular outcomes.

The baseline example remains dominated by freezing, whereas the three Hidden-Agenda examples develop different dominant win conditions during training.

The authors describe these patterns as basins of attraction in co-learning rather than Nash equilibria, where no player benefits from a unilateral change. Crewmate-dominated runs showed an earlier decrease in role-prediction loss, while losses became similar by the end of training; the comparison does not isolate the two added training components individually.

Role-prediction loss falls earlier in the two Crewmate-dominated groups than in the Impostor-freeze group, although the curves approach similar values later.

4.4 Partnered Crewmates also vote together

In 10 simulated episodes from a converged task-dominated group, the authors compared player distances and final-vote agreement after randomizing colors. Two nearby Crewmate pairs also tended to vote alike, supporting an association between proximity and voting similarity within this group.

The highlighted Crewmate pairs have relatively low within-pair distance and higher within-pair final-vote similarity in the analyzed episodes.

4.5 Voting round richness

The 25-step voting phase lets players change votes after observing others’ previous votes. In one post-hoc example, the group first voted out a Crewmate after an ambiguous attempted freeze, then voted out the Impostor after a second attempted freeze; the episode is available at https://youtu.be/cImPUBvmIq0.

The vote sequences show an incorrect first-round elimination followed by a second round in which enough players vote for the Impostor.

5 Conclusion

The authors conclude that hidden roles and unreliable information can produce partnering, voting, and opposing strategies in a spatial game without natural language communication. Their evidence concerns the reported game configuration and fixed co-trained groups.

Brief Thoughts

The baseline comparison and seed counts make the diversity of learned outcomes observable, but they do not establish which of the critic conditioning and auxiliary loss caused it. The single-seed parameter sweep and the 10-episode partnering analysis also limit confidence about robustness across settings and groups.