mechanistic interpretability

professionalexperimental

A research field that reverse-engineers the internal workings of artificial intelligence: it looks for which parts of the model produce the answer, and through what computations.

In plain terms

A language model is not written like a traditional program: it is trained on vast amounts of text, and its knowledge is encoded in billions of numbers. Even its creators cannot precisely explain why it gives the answer it does. Mechanistic interpretability attempts to open this "black box."

It is like a watchmaker taking apart an unfamiliar mechanism. It is not enough for them that the watch keeps accurate time; they want to know which gear drives which. Researchers similarly search for internal "components" in the model that carry individual concepts or steps of reasoning.

This is useful because it makes it easier to verify that the model gives a good answer for the right reason. The goal is to uncover hidden errors, biases, or deceptive behavior before the system influences important decisions. This is only partially successful today.

It differs from usual explanation methods in that it does not ask the model about its own reasons, nor does it merely look at the relationship between input and output. It examines the internal computation itself, as a neuroscientist studies how nerve cells work.

How it works

Researchers record the model's internal signals while it runs, meaning how active the artificial neurons are when processing a given text.
A neuron often responds to multiple, independent things, so auxiliary models (such as sparse autoencoders) break signals into cleaner, individually interpretable units called features.
They examine what triggers each feature: some activate on a city name, others on flattery or a software bug.
They map how features build on each other, meaning through which circuit the model travels from the question to the answer.
Finally, they verify by experiment: they artificially amplify or disable a feature and observe whether the model's behavior changes as expected.

An everyday example

You ask the chatbot why it rejected your request and receive a plausible explanation. However, this explanation does not necessarily match what actually happened inside the model. Using tools of mechanistic interpretability, the developer company's researchers can examine inside the model what led to the answer, though currently they only get a partial picture.

Who is affected

developerWith these tools, you can specifically identify what causes faulty or dangerous behavior, rather than just guessing blindly. → Code

lawyer, citizenIf an AI system makes a decision about you, the verifiability of that decision depends on how much insight we have into the model's operation. → Law

entrepreneur, executiveBefore entrusting a critical process to AI or an autonomous agent, you need to know how verifiable the underlying model is.

everyoneThe better researchers understand the internals of models, the more grounded your trust in everyday AI tools can be.

A common misconception

MythIf I ask the model why it answered as it did, I learn how it thought.

In realityA model's own explanation is text generated after the fact, which may differ from the true internal process. Mechanistic interpretability therefore measures internal computation and does not rely on the model's self-report.

Limits and risks

Current methods explain only a fraction of how the model works; we are still far from full understanding.
The interpretation of discovered features is a human reading that can be mistaken or oversimplified.
The analysis is very computationally and labor-intensive; for the largest models, it can currently only be performed partially.
If a company claims its model is transparent, that does not prove it is safe; ask exactly what was tested.

Similar, but not the same

explainable artificial intelligence – Explainable AI is a broader goal, often satisfied with providing understandable reasoning for a decision, whereas the mechanistic approach reveals the actual internal mechanism.
chain-of-thought – Reasoning is the text written out by the model, which may not faithfully reflect the internal computation, whereas interpretability examines this computation directly.
model evaluation (benchmark) – The evaluation measures what the model knows from the outside, while interpretability reveals how it reaches the result internally.

Related concepts

Broader

interpretability, explainable artificial intelligence, AI safety, transparency

Narrower

sparse autoencoder, feature, circuit, superposition, activation control

On the edge

large language model, Neural network, fekete doboz, alignment, hallucination

A brief history

The term was coined and popularized by researcher Chris Olah. In 2020, he and colleagues argued, in the article "Zoom In: An Introduction to Circuits" in the journal Distill, that the internal circuits of neural networks can be reverse-engineered. In 2023, Anthropic demonstrated on a small experimental model that neuron signals could be decomposed into interpretable features using a sparse autoencoder, and in 2024 it did the same on a production large language model. In 2025, it also published a method that maps the internal steps behind individual responses.

Where things stand

By the end of 2026, this is an active research area for major AI labs and university groups, whose tools are already appearing in pre-release safety checks of models. However, a full understanding of models remains a distant goal, and professional debate continues regarding the reliability of the methods.

Status: 2026 November

Also known as

mechanistic interpretability · mechanistic interpretability · mechanistic interpretability · mech interp · reverse engineering neural networks

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.