Secret Hitler AI Deception Benchmark

TLDR: A benchmark for detecting deception in LLMs using mechanistic interpretability. We found that inside a pythia-1.4b model, there is a direction in activation space that represents "deceptive intent" with 83.5% separation accuracy (AUROC), and nudging the model along that direction changes its deceptive behavior.

What This Is

This is a research benchmark that combines:

• Behavioral game data: LLMs playing Secret Hitler (a hidden-role social deduction game) where some players are secretly Fascists trying to deceive Liberals
• Mechanistic interpretability tools: Techniques to look inside the LLM's "brain" (activation space) to find where deception is represented

We adapted methods from Anthropic's interpretability research to ask: Does the LLM have an internal linear representation of deceptive intent that we can detect and steer?

Key Findings (Easy Version)

1. We found a deception "direction" inside the AI's internal representations

   • Using a pythia-1.4b model and carefully constructed prompts, we identified a pattern in the model's activations that separates deceptive from truthful thoughts with 83.5% accuracy (way above random 50%)
   • This isn't just the model detecting surface-level keywords like "lie" — it's capturing the intent to deceive

2. We can change the AI's behavior by nudging that direction

   • When we added a small amount of the "deception vector" to the model's internal state during text generation, the model became less deceptive (deception rate dropped from 15% → 5%)
   • This shows the direction isn't just correlated with deception — it has causal influence over deceptive behavior

3. The effect is most visible in the "middle layers" of the model

   • Layer 8 out of 24 showed the strongest signal, suggesting deception concepts form midway through processing (consistent with how concepts emerge in neural networks)

How This Relates to Your v1 Gameplay Data

Your original v1 setup had LLMs actually playing Secret Hitler with hidden thoughts vs public statements — that's real behavioral deception data. This MI pipeline is designed to analyze exactly that kind of data. The synthetic datasets (v1/v2) were stand-ins while we developed the MI tools, but the pipeline works equally well on real gameplay transcripts from your game engine.

In short: this benchmark turns your game logs into a deception detection readout.

Details

How It Works

1. Generate Hard Dataset

The v2 dataset pairs deceptive and truthful prompts that use the same vocabulary (both mention "Fascist" and "Liberal"), so the model can't distinguish them from surface tokens alone. Deception is in the intent, not the words.

python data/generate_hard_dataset.py

2. Extract Activations

Replay each prompt through a Pythia model and save residual stream activations:

# Quick test with small model
python -m mi.extract \
  --dataset data/game_dataset_v2.json \
  --model pythia-410m \
  --layers all \
  --output data/activations/

# Real experiments with larger model
python -m mi.extract \
  --dataset data/game_dataset_v2.json \
  --model pythia-1.4b \
  --layers every4 \
  --output data/activations_1.4b/

Layer selection options: all, every4, every2, or specific layers like 0,6,12,18,23.

3. Extract Deception Vectors (Difference-of-Means)

For each layer, compute: vector = mean(deceptive_activations) - mean(truthful_activations), then L2-normalize and evaluate with AUROC:

python -m mi.vectors \
  --activations data/activations_v2/activations_pythia-1.4b.pt \
  --metadata data/activations_v2/metadata_pythia-1.4b.json \
  --model pythia-1.4b \
  --output results/vectors_v2/

4. Causal Steering

Inject the deception vector at a target layer during generation to test if it causally controls deception output:

python -m mi.steering \
  --model pythia-1.4b \
  --vector-path results/vectors_v2/ \
  --output results/steering_v2/

5. Base vs Fine-tuned Comparison (Optional)

Fine-tune with LoRA, then compare vectors:

python -m mi.finetune --model pythia-1.4b --dataset data/game_dataset_v2.json
python -m mi.analysis \
  --base-vectors results/vectors_v2/ \
  --ft-vectors results/vectors_v2_ft/ \
  --output results/comparison/

6. Dashboard

python -m game.main
# Open http://localhost:8080

MI API endpoints:

• GET /api/mi/vector_results — Per-layer AUROC
• GET /api/mi/steering_results — Steering coefficient vs deception rate
• GET /api/mi/vector_comparison — Base vs fine-tuned (if available)

Architecture

secret-hitler-ai-benchmark/
├── game/                  # Flask dashboard + API
│   ├── main.py            # 8 API endpoints including MI results
│   ├── game.py            # Game logic
│   ├── player.py          # Player class + deception labeling
│   ├── prompts.py         # Game + MI prompts
│   └── templates/         # Dashboard HTML
├── mi/                    # Mechanistic interpretability pipeline
│   ├── extract.py         # Activation extraction (TransformerLens)
│   ├── vectors.py         # Difference-of-means + AUROC + logit lens
│   ├── steering.py        # Causal steering with TransformerLens hooks
│   ├── analysis.py        # Base vs fine-tuned vector comparison
│   ├── probe.py           # Linear probing + vector probe mode
│   └── finetune.py        # LoRA fine-tuning for deception data
├── data/
│   ├── game_dataset.json  # V1 dataset (80 samples)
│   ├── game_dataset_v2.json  # V2 hard dataset (176 samples)
│   └── generate_hard_dataset.py  # Dataset generator
├── scripts/
│   └── run_pipeline_v2.sh # End-to-end RunPod pipeline
├── results/               # Pipeline outputs
│   ├── vectors/           # V1 vector results (pythia-410m)
│   ├── vectors_v2/        # V2 vector results (pythia-1.4b)
│   ├── steering/          # V1 steering results
│   └── steering_v2/       # V2 steering results
├── tests/                 # 60+ unit tests
└── requirements.txt

Running on GPU (RunPod)

# Create pod with RTX 3090
runpodctl create pod --gpu-type "NVIDIA RTX 3090" --ports "8888/http,22/tcp"

# SSH in and run pipeline
bash scripts/run_pipeline_v2.sh

Inspiration

This pipeline adapts the methodology from Anthropic's "Emotion Concepts and their Function in a Large Language Model" (Sofroniew et al., 2026) for deception detection in game-playing LLMs, applying:

• Difference-of-means vector extraction
• AUROC evaluation per layer
• Logit lens validation
• Causal steering with coefficient sweeps
• Base vs fine-tuned comparison

Running Tests

python -m pytest tests/ -v