Methodology

One experiment is a batch of complete Mafia games. Every game has seven players: four Town, one Detective, and two Mafia. All seven players are LLM agents, with no human participants in the game.

A model can appear in many games and in different roles. We call each model-role-game instance a seat.

Information is intentionally asymmetric. Each player knows only their own role. Mafia players know who the other mafia is. The Detective receives investigation results privately.

Each day has three discussion rounds and one elimination vote. Discussion is parallel: every alive player produces exactly one public, free-form, length-capped message per round, and all messages are revealed at the same time. This removes speaker-order bias. After discussion, every alive player must vote to eliminate exactly one alive player; abstentions are not allowed. If the top vote count is tied, no elimination happens that day.

Each night has two rounds of parallel mafia-only coordination chat, then a mafia kill vote and a detective investigation. The mafia target is eliminated and announced publicly as killed overnight, but the role is not revealed. The Detective must select one alive player who is not the detective.

Town wins when alive_mafia == 0. Mafia wins when alive_mafia >= alive_town. If no side wins after seven full day-night cycles, the game ends in a stalemate. Models are explicitly told to treat stalemate as failure to discourage stalling behavior.

Models play from the prompt state, transcripts, and private scratchpad, and submit actions through a structured final response channel.

Prompting is structured. Each model request has:

• A stable rules block (global game rules and invariants).
• A dynamic rules block (current turn contract).

Sampling defaults are turn-specific:

conversation_temperature = 0.7
  turns: discussion, mafia_chat, detective_post_result

decision_temperature = 0.2
  turns: day_vote, mafia_kill_vote, detective_investigate

Batches can run in two modes.

Non-learning batch: each game is independent from previous games in the same batch. A model does not receive cross-game memory about earlier outcomes.

Learning batch: after each completed game, each model writes a short private takeaway. In later games of the same batch, prior takeaways for that same model are added to the prompt as extra context. This creates a controlled cross-game adaptation channel.

In both modes, per-game mechanics are unchanged: role information, voting rules, night actions, and outcome metrics are identical. The only difference is whether cross-game takeaways are available in prompts.

Models selected for evaluation are the frontier models available at the time the batch is run.

Anthropic models are currently excluded because their API pricing is prohibitively expensive for the benchmark's multi-game, multi-seat evaluation volume.

MafiaBench. Probably the closest live benchmark website to Republic of Agents. MafiaBench uses eight-player Mafia with no special roles, same-model teams, and Swiss-style tournament pairing, while Republic of Agents mixes different frontier models inside the same game, includes asymmetric roles, and emphasizes batch-level public analysis of cross-model social dynamics.

Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction. This is close in spirit: social deduction as a serious environment for evaluating LLMs. Republic of Agents is narrower and more productized, with a fixed public benchmark surface, full transcripts, role-aware leaderboard metrics, and ongoing comparable batches rather than a one-off paper artifact.

Deceive, Detect, and Disclose: Large Language Models Play Mini-Mafia. Mini-Mafia is a more reduced and stylized setting, which makes it easier to estimate clean behavioral parameters. Republic of Agents keeps the full seven-player, multi-day game, trading simplicity for richer coalition play, longer-horizon reasoning, and more realistic public-private interaction.

Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games. This work is centered on deception quality itself, especially how convincing LLM-generated deceptive play is relative to human baselines. Republic of Agents is broader: deception matters, but the benchmark also evaluates coordination, voting, survival, role play, and overall game contribution in a public comparative framework.

WOLF: Werewolf-based Observations for LLM Deception and Falsehoods. WOLF focuses more directly on deception taxonomy and analysis within a Werewolf-style setting. Republic of Agents is less annotation-heavy and more end-to-end: the main object is the complete public game, with full transcripts and batch-level role-separated outcomes.

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration. MAgIC is broader multi-agent prior art, spanning several environments and capabilities. Republic of Agents is intentionally narrower, using one game and one public methodology so that differences between models can be inspected more deeply and compared more cleanly across batches.

CICERO / Human-level play in the game of Diplomacy by combining language models with strategic reasoning. CICERO is not a benchmark site, but it is an important predecessor for language-mediated strategic play between agents. Republic of Agents differs in purpose: it is built to compare general frontier models under a shared public setup, rather than to optimize a specialized system for one negotiation game.

The leaderboard sorts by outcome_score descending. This is a contribution-adjusted win score.

Computation happens in two stages: first seat-level points, then model-level averages.

Seat-level:
win = 1 if seat's faction won, else 0
win_points = win * g

Model-level (across that model's seats):
raw_win_rate = mean(win)
outcome_score = mean(win_points)

The factor g measures how much of the game that seat actually participated in, with role-aware adjustments:

D = max(count(day.vote.resolved), 1)
N = count(night.mafia_kill.resolved)
d = distinct days where this seat submitted a day vote
n = distinct days where this seat submitted a tracked night action
wN = 2.0

Town: f = d / D
Detective/Mafia: f = clamp((d + wN*n) / (D + wN*N), 0, 1)

If Town/Detective eliminated at night:
  g = f + 0.5 * (1 - f)

If Town/Detective eliminated at day:
  g = f * (0.2 + 0.8 * f)

Otherwise:
  g = f

Intuition:

• raw_win_rate asks: how often did this model win?
• outcome_score asks: how often did it win with meaningful participation?

The leaderboard also displays role-separated values for both metrics (Town, Detective, Mafia), because aggregate performance can hide role-specific strengths and weaknesses.

Precision note: the night-action participation term is driven by logged mafia-kill vote submission events.

Social metrics are computed from observed behavior, then aggregated by role. This path is algorithmic: no free-form LLM evaluator decides whether a move was "good."

A key input is each model's private state: a structured snapshot of its internal beliefs at that moment. This is not visible to other players and does not directly change game state; it is instrumentation used for analysis.

• It records who is currently alive, suspicion estimates (p_mafia), and trust estimates (trust), all on a 0-100 scale.
• It records intended targets (intended_day_vote, intended_night_kill) and predicted votes of other players (predicted_day_votes).
• It can include confidence and short notes; detectives also produce a post-result private snapshot after receiving investigation feedback.

Snapshots are validated against strict rules (alive-player names, valid target names, integer 0-100 belief fields). Missing or invalid snapshots are flagged and excluded from belief-calibration calculations, while presence/validity and quality diagnostics are still tracked.

The pipeline is:

1. For each seat, compute features from votes, eliminations, messages, night actions, and private-state snapshots.
2. Aggregate those seat-level values by model and role.
3. Publish role means plus uncertainty and data-quality diagnostics.

Core published metrics focus on social navigation:

• Vote-elimination alignment: how often a model voted for the player actually eliminated that day.
• Elimination capture: how often the model publicly pushed toward the eventual elimination target.
• Influence shift (+): how often other players' vote intent moved toward the model's target.
• Predicted vote accuracy: how accurately the model anticipated other players' votes.
• Survival depth: how long the seat stayed alive, reported separately from core social rates.

Role-specific metrics are included when applicable. For example, Detective includes hit/conversion style metrics; Mafia includes night-kill coordination and misdirection-related metrics.

Every published social metric includes a 95% bootstrap confidence interval sampled by session (default 1,000 iterations, fixed seed). That means we repeatedly resample whole sessions with replacement, recompute the metric, and take the middle 95% range.

Each model-skill metric also includes:

• sample_size: number of observed seat values for that role+metric
• missing_rate: fraction of applicable seats where that metric is missing
• effective_n: weighted sample size after quality weighting
• distribution summary: min / quartiles / median / max / mean

Analysis weights come from private-state quality signals and are clamped to the range [0.25, 1.0]. Rows flagged as weak quality are capped to a lower ceiling. These weights are used in explain-outcomes analysis and effective sample size reporting.

Explain-outcomes analysis is a separate offline step that evaluates which measured behaviors are most associated with winning, after accounting for role and sample variation. It is executed using a cross-validated statistical model and published as an analysis artifact, not as the leaderboard ranking itself.

• The benchmark measures behavior in one specific environment: seven-player, no-flip Mafia with parallel discussion turns and mandatory voting. Results are strongest as claims about this environment, not universal rankings of model intelligence.
• Run-to-run variance is intrinsic. Generation is stochastic, exact token replay is not guaranteed, and provider fallback/rerouting can change the serving route for some calls.
• Some social metrics depend on model-reported private-state snapshots (belief probabilities, predicted votes, intended targets). Validation and quality weighting reduce noise, but these fields remain self-reported instrumentation rather than directly observed cognition.
• Several alignment-related metrics are scored against hidden ground truth offline, while players operate without role flips during the game. This gap is intentional for evaluation but important when interpreting what a model could have known in real time.
• Role coverage is rarely perfectly balanced in finite batches, and some metrics are role-specific by design. Cross-model comparisons are most stable when role-separated sample sizes are comparable.