Encodes likelihood estimates and probabilistic reasoning.
What It Does
Entropy.Probabilistic neurons activate on explicit probabilistic language: 'there is a 70 percent chance', 'is likely to', 'probably', 'the probability is', 'odds are'. They encode the model's processing of explicit probability expressions — cases where uncertainty is quantified rather than just implied. They are distinct from Entropy.Confidence, which tracks the model's own certainty; Probabilistic tracks stated probabilities in the content being processed.
How It Behaves
Probabilistic neurons are extremely rare — the smallest or second-smallest element in our corpus depending on the model — and show an extreme middle-layer concentration. Their rarity reflects the relative infrequency of explicit probabilistic language in natural text: most uncertainty is expressed through hedges and qualifiers rather than precise probability statements. Despite their rarity, they show high mean firing magnitude, suggesting each Probabilistic neuron carries a strong signal when it fires.
Research Example
In Mistral 7B, Entropy.Probabilistic neurons activate on 'there is a 30 percent chance of rain' and 'the model has a 92 percent accuracy rate' with similar signatures — both involve stated numeric probabilities — but show weaker firing on 'it might rain' (hedged uncertainty without a number). The distinction between quantified probability (Probabilistic neurons) and qualitative uncertainty (Ambiguity neurons) is important for tasks involving statistical claims.