Round 5 · Grassmannian attention — diagnosing the 2-hop problem
The 2-hop recall failure of Grassmannian attention (k=2) could have two explanations: either the subspace dimension k=2 is just too small for the task, or the 2-hop problem has a fundamentally different bottleneck that subspace expressiveness can't address.
This experiment tests the first explanation directly: does increasing k — using 3D, 4D, or 8D subspaces instead of 2D — improve 2-hop recall performance?
If you're struggling to parallel park because your car's turning radius is too small, getting a car with a tighter turning radius (more k) helps. But if you're struggling because the parking space is too narrow regardless of turning radius, a better car won't help. This experiment determines which problem applies.
2-hop flat associative recall with Grassmannian attention at k=1 (equivalent to dot-product), k=2, k=3, k=4, and k=8. All other parameters held constant. 10K training steps.
| k | Accuracy at 10K steps |
|---|---|
| 1 (dot-product) | ~20% |
| 2 | 17.9% |
| 3 | ~19% |
| 4 | ~19% |
| 8 | ~19% |
All Grassmannian variants are tied at ~19%, essentially the same as standard attention (k=1). Increasing k from 2 to 8 produces no meaningful improvement.
The 2-hop recall bottleneck is not subspace expressiveness. Even with an 8-dimensional subspace comparison per attention pair, performance doesn't improve. The problem lies elsewhere — specifically in cross-layer coordination.
For 2-hop recall, the model needs to: (1) match query A to the correct key in one attention head, (2) extract the associated value B, and (3) route B to a second attention head that uses it as a new query. This is a multi-layer, multi-step coordination problem. More expressive similarity at each step doesn't help if the bottleneck is how information flows between steps.
This is a useful diagnostic: it rules out the "not enough geometric expressiveness" explanation and points toward the "hard optimisation problem of cross-layer chaining" explanation. A scratchpad mechanism — explicitly writing the intermediate value to a buffer position that a second attention layer can directly attend to — would address this at the architectural level.
- Does explicit intermediate state (a scratchpad token) enable reliable 2-hop recall for both standard and Grassmannian attention?
- Is the cross-layer coordination bottleneck fundamental to attention, or is it an optimisation difficulty that vanishes with more training?