Round 4 · p-adic geometry
p-adic numbers define a type of geometry called ultrametric geometry, which is the natural geometry of trees: in a tree, the distance between two nodes is determined by the depth of their nearest common ancestor, not the sum of individual path lengths. This has a distinctive property — the "triangle inequality" becomes much stronger. Could attention mechanisms exploit this geometry to better represent hierarchical structure?
Before testing this in attention, we needed to answer a more basic question: can a standard Euclidean embedding even represent ultrametric tree distances faithfully? If it can, there's no geometric argument for p-adic representations.
Imagine trying to draw a family tree on a flat piece of paper, where physical distance on the page should represent how closely related people are. Is it even possible to arrange everyone so that the distances work out correctly? This experiment tests the 2D equivalent of that question for binary trees in high-dimensional Euclidean space.
We generated binary trees of various sizes and trained Euclidean embeddings to reproduce their pairwise ultrametric distances using cosine similarity. We measured the distortion — how much the learned embeddings deviate from the true tree distances.
At 32 leaves in 32-dimensional space (d = N), learned Euclidean embeddings converge to near-zero distortion (MSE → 0 at step 1,000). The embedding problem is perfectly solvable.
A hierarchical path-encoding initialisation (starting with embeddings that explicitly encode tree position) caused convergence that was 4× slower than random initialisation.
Euclidean space can represent ultrametric tree distances without distortion when d ≥ N. There is no irreducible geometric gap that would force you to use p-adic or hyperbolic representations at these scales. The standard embedding space is sufficient.
The initialisation finding is equally important: providing embeddings with "built-in" geometric structure from the start hurts convergence, not helps it. Gradient descent is better at finding the right embedding layout when it starts from scratch in a high-dimensional space, without a pre-imposed geometric constraint. Geometric priors, when imposed before training, constrain the optimiser's search space in ways that slow it down.
This result ruled out p-adic geometry as a necessary architectural component for hierarchical tasks at practical scales. The performance gap that exp-depth-sweep reveals is not because the embedding space can't represent the structure — it's because the learned attention mechanism fails at the phase transition boundary.
- Does this result hold at very large N (thousands of leaves) where the Euclidean embedding might become underdetermined? That would be the regime where hyperbolic geometry's advantages (exponentially expanding volume) could matter.