Encodes programming language constructs and syntax.
What It Does
Symbol.Code neurons activate on programming language syntax: variable declarations, function signatures, control flow constructs (if/else, for loops, while), operators, brackets and parentheses in code contexts, and language keywords (def, class, return, import). They are distinct from natural language Symbol neurons in that code symbols carry precise semantic weight — the order and structure of code symbols is syntactically binding in ways that natural language is not.
How It Behaves
Code neurons are among the most strongly activated Symbol sub-types and show broad layer distribution. They are more prevalent in models with significant code training data (Llama 3.1, Qwen3, Mistral) and less so in models trained primarily on natural language (Gemma 2B, early GPT-2 variants). A distinctive property: Code neurons activate on code regardless of programming language — Python, JavaScript, and Rust trigger similar Code neuron patterns despite their syntactic differences, suggesting the model has learned a language-agnostic 'code structure' representation.
Research Example
In Llama 3.1 8B, Symbol.Code neurons fire equally on equivalent implementations in Python (`def add(a, b): return a + b`) and JavaScript (`function add(a, b) { return a + b; }`). The abstract structure — named function, parameters, return statement — produces similar neuron firing despite completely different surface syntax. This is why large language models can translate between programming languages: the Code neuron representation is partially language-agnostic.