Round 6 · Confirming the mechanism
Grassmannian attention uses a k×k inner product matrix per attention pair, which means — naively — it has more parameters in the Q and K projections than standard attention at the same d_head. Could the depth-6 advantage simply be explained by having more parameters, rather than by the geometric properties of subspace comparison?
This is a critical sanity check. Science requires ruling out simpler explanations before claiming a novel geometric effect.
Suppose a larger-engined car beats a smaller-engined car in a race. You can't conclude that engine design matters until you also compare two cars with the same engine size but different designs. This experiment is that comparison: same Q/K parameter count, different similarity primitive.
At tree depth 6 (the phase transition point), we ran three variants:
- Standard attention — d_head=16, Q/K params = 16,384
- Grassmannian k=2, parameter-matched — d_head=8 for Q/K, Q/K params = 16,384 (identical to standard)
- Grassmannian k=2, extra params — d_head=16, Q/K params = 32,768 (2× standard)
| Model | Q/K params | Total params | Final accuracy (5K steps) |
|---|---|---|---|
| Standard attention | 16,384 | 107,872 | 23.9% |
| Grassmannian (matched) | 16,384 | 99,680 | 97.9% |
| Grassmannian (extra) | 32,768 | 124,256 | 98.5% |
The parameter-matched Grassmannian — using fewer total parameters than standard attention — achieves 97.9% vs 23.9%. The advantage is geometric, not parametric.
The subspace overlap primitive provides fundamentally more geometric expressiveness than the dot-product primitive at equal parameter cost. With the same number of weights to learn, the k=2 Grassmannian similarity can solve a problem that k=1 dot-product cannot.
The intuition: with 64 leaves and 64 dimensions, each token must simultaneously "point toward" all its tree-neighbours and "point away from" all non-neighbours. A 1D projection (dot-product) can satisfy one such constraint at a time. A 2D subspace projection has the geometric degrees of freedom to satisfy more constraints simultaneously — enough to solve the packing problem at d=N.
The parameter-matched Grassmannian also has fewer total parameters (because d_head=8 rather than 16 for the value and output projections), yet outperforms standard attention dramatically. This is a clean geometric story: better primitive, same budget, much better result.
The extra-param Grassmannian is only marginally better (98.5% vs 97.9%), confirming the advantage comes from the k=2 subspace structure, not from scale.
- Does the parameter-matched Grassmannian also succeed at depth 7? Or does it share standard attention's failure at d/N < 1?
- At what d/N ratio does even Grassmannian fail — what is the geometric expressiveness limit of k=2 subspace comparison?