LLM Cross-Replay Decomposition

This repository accompanies the paper What Do Transformer Activation Measurements Capture During Language Generation? A Cross-Replay Decomposition, by H. Ugail, N. Howard, and I. Mehmood.

Hidden activations recorded while a transformer language model generates text are often analysed as if they were intrinsic signatures of the model. The difficulty is that, during generation, an activation depends both on the model and on the text the model has just produced, so it is not obvious what an activation measurement actually captures. Cross-replay is a control paradigm that separates these two sources of variation. Sequences generated by one variant of a model are replayed as fixed token sequences through several variants of the same base model, so every replay processes identical text, and the only thing that differs across replays is the replay variant.

What does this measure?

For each activation measurement, cross-replay combined with a regression on surface text statistics partitions the variance into:

1. A measured-text-associated component — variation explained by surface statistics of the generated continuation (token entropy, distinct n-gram ratios, repetition rate, mean log token frequency, intact-model perplexity, and so on).
2. A variant-associated component — the incremental variation attributable to the replay variant or perturbation identity once the measured text statistics and design controls have been accounted for. The manuscript writes this incremental variance as $\Delta R^{2}_{\text{variant}}$, which appears in the saved tables as the dR2_variant column.

Across four decoder-only transformer models — GPT-2-medium, Pythia-410M, Qwen2.5-0.5B, and GPT-J-6B — the headline pattern is:

• Temporal and spectral summaries of replayed activation trajectories (DFA exponent, autocorrelation time, spectral slope, effective rank, trajectory velocity) carry little incremental variant-associated signal once measured text statistics and design controls are included. Across the twenty temporal/spectral cells, $\Delta R^{2}_{\text{variant}}$ ranges from $0.003$ to $0.032$.
• Representational-geometry and attention-divergence measurements (CKA distance, Procrustes distance, attention Jensen–Shannon divergence) retain a substantially larger perturbation-sensitive component under fixed-text replay.

Pythia-410M is the one documented exception: its CKA and Procrustes cells have leave-one-prompt-out text-only cross-validated $R^{2}$ of $0.194$ and $0.237$, which exceed the corresponding $\Delta R^{2}_{\text{variant}}$ of $0.101$ and $0.138$. This shows that, even under fixed-text replay, geometry measurements can retain a measurable association with generated-text properties in some model families.

Contents

• Crossreplay_Decomposition.ipynb — the full analysis pipeline. Runs the generation phase, the replay phase, all nine activation measurements (five temporal/spectral, three primary pairwise, and one exploratory probe), the variance decomposition with text-statistical and design controls, the prompt-cluster bootstrap, the design-respecting structured permutation negative control, cluster-robust standard errors, the mixed-effects sensitivity diagnostic, the matched-feature text-statistical robustness check, the extended text-control sensitivity analysis, the multiple-comparison summary, and the publication figures.

• verify_results.py — a quick verification script for reviewers. Loads the precomputed result tables in Results/ and confirms that every headline number reported in the paper is reproducible from those tables. Takes a few seconds, needs no GPU, and prints a clean PASS/FAIL summary.

• Results/ — the precomputed result tables and figures produced by the notebook for the four models reported in the paper, sufficient to reproduce every figure and table in the manuscript without re-running the pipeline. The directory contains the cross-family master variance-decomposition table, the structured-permutation negative-control summary and full per-permutation values, the multiple-comparison and effect-size tables, the matched-feature text-prediction robustness file, the extended text-control sensitivity table, the perturbation-severity summary, the mixed-effects sensitivity diagnostic, and the four publication figures.

• requirements.txt — full dependencies for re-running the notebook (PyTorch, Hugging Face Transformers, accelerate, etc.). A CUDA GPU is recommended.

• requirements-verify.txt — minimal dependencies for verify_results.py only (pandas, numpy, statsmodels). No GPU needed.

Quick start

The fastest way to check the headline numbers is:

git clone https://github.com/ugail/LLM-Cross-Replay-Decomposition.git
cd LLM-Cross-Replay-Decomposition
pip install -r requirements-verify.txt
python verify_results.py

Reproducing the full pipeline

Re-running the full pipeline requires a CUDA GPU. A single 16 GB GPU is sufficient: GPT-J-6B runs in fp16, the three smaller models in fp32. The notebook is the same one used to produce the precomputed tables in Results/. It runs one model family at a time. Set MODEL_NAME in the configuration cell to one of:

• "gpt2-medium"
• "EleutherAI/pythia-410m"
• "Qwen/Qwen2.5-0.5B"
• "EleutherAI/gpt-j-6B"

and execute the cells in order. The notebook autodetects whether it is being run in Google Colab or locally. In Colab, results are written to a fixed Google Drive folder under the configured project root; locally, they are written to ./Results/ in the working directory. Once all four families have been processed, the cross-family analysis cells at the bottom of the notebook combine the per-family outputs into the master table, the negative-control summary, the multiple-comparison and effect-size tables, the matched-feature text-statistical robustness file, the extended text-control table, and the four publication figures.

A full run for all four models takes roughly an hour on a 16 GB GPU.

Who is this for?

• Researchers in interpretability who want a diagnostic control for activation measurements during generation.
• Researchers in representational similarity who want a worked example of how text-statistical and replay controls behave for CKA distance, Procrustes distance, and attention divergence side by side.
• Methods builders who want a reusable template for variance decomposition with prompt-cluster bootstrap intervals, structured permutation tests, cluster-robust and mixed-effects sensitivity diagnostics, extended text-control robustness, and multiple-comparison correction.

Citation

If you use the cross-replay paradigm, the code in this repository, or the precomputed result tables, please cite:

H. Ugail, N. Howard, and I. Mehmood. What Do Transformer Activation Measurements Capture During Language Generation? A Cross-Replay Decomposition. Under review.

License

The code in this repository is released under the MIT License. See LICENSE for the full text.