Encodes similarity and analogical reasoning relationships.
What It Does
Relation.Analogy neurons activate on analogical and similarity relationships: explicit analogies ('X is like Y', 'A is to B as C is to D'), metaphorical mappings ('the brain is a computer'), comparative similarity ('functions similarly to', 'operates in the same way as'), and structural parallels ('the same pattern appears in'). They encode the relationship of structural or functional similarity between two domains.
How It Behaves
Analogy neurons are among the smallest elements in the corpus and show an extreme early-layer concentration — nearly all Analogy neuron activity happens in the first few layers. This is one of the most counterintuitive findings in our research: analogical reasoning, which humans experience as a high-level cognitive achievement, appears to be grounded in early-layer pattern matching in transformers. The implication is that analogical failures in language models may be architectural — occurring before deep contextual processing can occur — rather than a failure of high-level reasoning.
Research Example
In Llama 3.1 8B, Relation.Analogy neurons fire on 'neural networks learn like children — through exposure and feedback' in the very first processing layers, before the model has fully integrated the meaning of either 'neural networks' or 'children.' The analogy frame is established early, constraining how the subsequent content is interpreted. This early-frame establishment is why analogies can be both powerful (they organize understanding efficiently) and dangerous (they can lock in a frame that distorts interpretation of evidence that doesn't fit the analogy).