Open Problems in Mechanistic Interpretability | Summary

This article explains the key points of Open Problems in Mechanistic Interpretability.

• 2025-01-27 (arXiv)
• Sharkey, Lee, Chughtai, Bilal, Batson, Joshua, Lindsey, Jack, Wu, Jeff, Bushnaq, Lucius, Goldowsky-Dill, Nicholas, Heimersheim, Stefan, Ortega, Alejandro, Bloom, Joseph, et al.
• Apollo Research, Anthropic, Decode Research, Eleuther AI, FAR AI, Google DeepMind, Leap Laboratories, Harvard University, King’s College London and Imperial College London, MATS, MIT, METR, Northeastern University, Pr(AI)²r group, Tel Aviv University, Timaeus, University of Melbourne, Goodfire
• Paper

Read this article in Korean

Summary

• Mechanistic interpretability seeks to explain the internal computations that let neural networks generalize beyond their training examples. This review finds that current methods offer partial insights but have not shown consistent practical usefulness in evaluations against relevant alternatives that do not require interpretability; it synthesizes prior research rather than reporting a new controlled comparison.
• The review organizes methodological work around decomposing networks, describing component functions, and validating those descriptions. Sparse dictionary learning (SDL), which represents hidden activations—internal numerical states—using learned dictionary elements with few active at once, receives particular scrutiny because reconstruction errors, uncertain assumptions, and dependence on training data limit what its components establish about computation.
• Applications include monitoring artificial intelligence (AI) systems, controlling behavior, predicting failures and capabilities, improving training and inference, and extracting scientific knowledge. The authors call for stronger tests of causal explanations and fair comparisons on practical tasks, while warning that selective transparency can create false assurances about safety.

1 Introduction

Neural network capabilities emerge through training even when developers do not understand the mechanisms implementing them. The introduction defines understanding operationally: knowledge of internal mechanisms should enable successful behavioral predictions or practical interventions, such as more precise control or improved network design.

• Scientific motivations include understanding artificial cognition and recovering patterns learned by models in domains such as protein structure prediction and medicine.
• These benefits are research objectives whose reliable realization remains an open problem.

1.1 The focus of this review: Open problems and the future of mechanistic interpretability

The review examines the field’s current methods and which unresolved problems deserve priority. Under “1.1.1 Why ‘mechanistic’ interpretability?”, it distinguishes interpretable model design, explanations of individual decisions, and explanations of mechanisms shared across a general class of problems.

• The paper uses mechanistic interpretability specifically for research into mechanisms underlying neural network generalization.
• Its motivating hypothesis is that generalization arises from shared computation, making recurring internal processes a useful object of study.

1.2 Types of open problems

The open problems concern methods and foundations, applications, and the social context surrounding technical work. Desired outcomes include detecting dangerous internal processes, modifying behavior, predicting behavior in unfamiliar situations or the development of capabilities, improving computation, and extracting learned knowledge.

• The authors argue that wider application of existing tools must be accompanied by methodological improvements.
• Different applications require different kinds and degrees of understanding, motivating the later analysis of progress along multiple axes.

2 Open problems in mechanistic interpretability methods and foundations

The methods discussion treats networks as learned algorithms that transform internal activations to produce outputs. It distinguishes reverse engineering, which starts with network components and investigates their roles, from concept-based interpretability, which starts with proposed concepts and searches for corresponding components.

• Figure 1 illustrates these two starting points.
• Combining tools into reliable research procedures and automating those procedures are separate challenges that affect both approaches.

The panels show how the starting point changes the investigation. Reverse engineering seeks functions for decomposed network components, including unfamiliar functions; concept-based analysis searches for components corresponding to concepts already specified by the investigator.

2.1 Reverse engineering: Identifying the roles of network components

Reverse engineering is needed because similar task performance does not imply similar computation in humans and models. Examples include image models relying on texture or incidental dataset cues, and a small transformer, a network architecture using attention to combine information across positions, solving modular addition through a Fourier transform strategy that represents the problem in terms of frequencies.

• The workflow has three steps: decomposition into simpler parts, hypotheses about component functions and interactions, and validation.
• Figure 2 makes revision explicit: failed validation should lead to changes in the decomposition or the functional hypothesis.

The return arrows make validation part of an iterative process. A failed prediction can expose either an unsuitable decomposition or an incorrect description, requiring revision before the components’ roles can be considered understood.

Architectural units are imperfect candidates for functional units. Neurons and attention heads, the parallel components that select and combine information in a transformer, can be polysemantic, meaning they respond to multiple kinds of features; representations can also extend across layers.

• Principal Component Analysis, Singular Value Decomposition, and non-negative matrix factorization identify patterns in hidden activations through lower-dimensional or factored representations, but that structure need not match computational structure.
• The superposition hypothesis proposes that sparsely active features can outnumber activation dimensions; ordinary dimensionality reduction cannot recover more directions than there are dimensions.

SDL attempts to recover an overcomplete dictionary, with more learned directions than activation dimensions. Sparse Autoencoders (SAEs), encoder-decoder networks trained to reconstruct activations using sparse internal codes, are one implementation; transcoders reconstruct next-layer activations, and crosscoders can reconstruct activations across multiple layers.

• A latent is a learned dictionary element whose activation indicates its contribution to a reconstruction.
• Figure 3 connects this approach to the linear representation hypothesis: concepts are represented by vectors whose addition and scaling express composition and intensity. The paper distinguishes a weaker claim about some concepts from a stronger claim about all concepts, which cited work refutes for some models.

The diagram links linear concept representations, sparse sharing of a limited activation space, and recovery of more dictionary elements than activation dimensions. Together, these ideas motivate SDL while exposing the assumptions its interpretation depends on.

Reconstruction substitution tests SDL by replacing original model activations with dictionary reconstructions and measuring the resulting performance. The review reports that inserting a dictionary with 16 million latents into GPT-4 produced language modeling loss equivalent to a model with only 10% of GPT-4’s pretraining compute; a separate GPT-2 small study reported performance decreases of 10% with task-specific SAE training data and 40% with full-distribution training data.

• These figures come from cited studies with different conditions. The GPT-4 comparison concerns prediction loss, not the compute used to train or run the substituted model.
• Other limitations include computational cost, unresolved geometric relationships between feature directions, difficulty handling representations across heads and layers, and the failure of sparsity to consistently track interpretability. Figure 4 collects these limitations.
• SDL identifies activation directions rather than directly describing the mechanisms implemented by network parameters. Its learned concepts also depend on training data: cited SAEs trained on pretraining data generally lacked good refusal-related latents, whereas SAEs trained on chat-formatted data had them.

The surrounding panels show why reconstruction quality alone cannot establish mechanistic understanding. Better reconstruction can still leave unresolved which concepts were recovered, whether their geometric relationships matter, and how the network transforms activations.

The theoretical problem extends beyond improving reconstruction: the field lacks an agreed formal definition of features or canonical units for decomposition. The review proposes investigating feature geometry, computation in superposition, causal foundations, and connections to theories of generalization.

• Intrinsic interpretability means encouraging easier interpretation during model construction or training, using discrete codes, sparse activations or weights, modularity, or mathematically tractable architectures.
• The paper notes that many attempts to directly encourage interpretable activations have not maintained competitive performance and have not eliminated superposition.
• Even an intrinsically decomposable model still requires functional descriptions and validation.

Component descriptions should explain what causes activation and what activation subsequently changes. Highly activating examples and feature synthesis, which generates inputs that maximize component activation under constraints, can suggest hypotheses, but dataset bias and human expectations can produce convincing yet misleading interpretations.

• Attribution methods estimate the influence of upstream variables on downstream quantities; gradient approximations can miss nonlinear effects, while perturbations can move the model outside its training distribution.
• The logit lens maps intermediate representations through the model’s output mapping into vocabulary predictions, exposing direct output effects but missing effects mediated through later layers.
• Causal interventions replace internal values during a forward pass and observe changes; activation patching replaces activations, while path patching isolates particular connections. Figure 5 distinguishes upstream causes from intermediate and final effects.
• Patchscopes transfer activations into another forward pass whose context prompts a language model to reveal information encoded in them. Reading chain-of-thought, a model’s generated reasoning steps, does not directly inspect internals, and cited studies show that these steps need not faithfully describe the underlying computation.

The blue component lies between upstream causes and downstream effects, with intermediate components distinguished from inputs and outputs. This separates two explanatory tasks: identifying what activates a component and predicting what its activation changes elsewhere in the network.

Validation asks whether a functional hypothesis makes accurate predictions about activations, interventions, failures, or useful changes to behavior. The review recommends multiple tests, including handcrafted replacements, networks with known computations, and comparisons against relevant engineering baselines.

• “Model organisms” are extensively studied reference networks that support cumulative experiments and cross-checking. Desired properties include open training data, reproducible training, low cost, broad representativeness, and instances trained with different random seeds.
• Small reference models offer tractability but can encourage methods that do not generalize to practical safety settings.
• Benchmarks can use programs compiled into network weights or training procedures that enforce known explanations. Predictive and practical validation remain necessary because plausible descriptions and some existing metrics can be misleading.

2.2 Concept-based interpretability: Identifying components for given roles

A concept-based probe is a classifier trained to predict a specified concept from another model’s hidden activations. Probes require carefully defined labels and can recover correlations even when the identified activations do not causally affect the output.

• Contrast-Consistent Search seeks an activation axis corresponding to positive and negative propositions through probabilistic consistency, but still requires suitably constructed contrasting examples. Standard probes are similarly limited to concepts defined well enough to label.
• Counterfactual data, which varies a concept of interest, and interventions can test causal relevance; concept erasure tests the effect of removing concept information from representations. Such methods require more specialized data, and evaluation on unfamiliar distributions is needed to detect spurious correlations.
• Concept bottleneck models incorporate specified concepts during training, increasing the prospect that assigned components are causally relevant. Networks can also be trained to use specified causal structures, although prespecifying every useful concept or structure remains difficult.

2.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines: A case study

Circuit discovery identifies a task-relevant subgraph, or circuit, of interacting model components. Its typical pipeline defines a task, chooses components, identifies relevant nodes and connections through interventions, iterates between descriptions and tests, and performs final validation.

• Faithfulness measures agreement with the full network’s behavior, minimality checks for unnecessary included components, and completeness checks whether important components were omitted.
• Human-defined tasks can hide substantial variation between individual examples, and measured faithfulness can depend on how interventions are implemented.
• Scalable gradient approximations, backup computation that compensates for interventions, negative behavior that suppresses task performance, and selection of unusually simple tasks all limit current pipelines.

2.4 Automating steps in mechanistic interpretability research

Automated interpretation can generate descriptions from strongly activating examples and test those descriptions by predicting activation patterns. Automated Circuit DisCovery (ACDC), a method for finding task-relevant computational subgraphs, automates another part of the workflow.

• Finding a subgraph does not automatically explain the functions of its components.
• The authors argue that fully automating current procedures would retain their methodological weaknesses and would not yet produce satisfactory explanations.
• Proposed improvements include generated test inputs, stronger statistical tests, and descriptions of feature effects and interactions.

3 Open problems in applications of mechanistic interpretability

The applications discussion asks which technical advances are needed for concrete scientific and engineering objectives. Figure 6 maps monitoring, control, prediction, improved computation, scientific discovery, broader model coverage, interfaces, and governance.

• These areas have different requirements, so improvement in one aspect of interpretability need not advance every application equally.
• The review also anticipates that internal analysis could reveal problems not identified through existing behavioral evaluations.

The eight rows organize application areas rather than report performance. They show the range of objectives against which interpretability must be assessed, including monitoring, control, scientific discovery, interfaces, and governance.

3.1 Axes of mechanistic interpretability progress.

The paper separates progress in decomposition from progress in the depth of component descriptions. Additional axes concern how much of the network is understood, how broad a task distribution is covered, and whether analysis explains a fixed model or changes during training.

• Monitoring one behavior may require only a narrow subset of components and inputs.
• Enumerative safety proposes identifying all relevant mechanisms to assess whether dangerous ones are absent. This and formal verification, mathematically proving specified behavioral properties over a defined input distribution, require broader coverage.

3.2 Using mechanistic interpretability for better monitoring and auditing of AI systems for potentially unsafe cognition

White-box evaluations inspect model internals for concerning patterns that input-output testing might miss. Examples motivating this work include sandbagging, deliberate underperformance during evaluation, and sycophancy, responses tailored dishonestly to match a user’s beliefs.

• Correlated internal indicators may help evaluators formulate and investigate concerns without a complete causal account or understanding of the entire network.
• The unresolved coverage problem is whether learned features capture all relevant concerning reasoning. A feature that recognizes deception must also be distinguished from a mechanism that generates it.

Deployment monitoring could inspect internal states continuously or flag unusual reasons for an action. Mechanistic anomaly detection, the proposed detection of atypical internal mechanisms, requires a clearer account of mechanisms than identifying active dictionary latents alone provides.

• Red-teaming means deliberately seeking failures or unsafe outputs. Cited work uses a refusal direction, an activation direction causally involved in refusing harmful requests, to guide adversarial input search.
• The review suggests that this targeted search may be more efficient than optimizing over the whole model; it reports no new measured speedup.

Activation steering changes behavior by adding a fixed vector to intermediate activations during inference. Machine unlearning seeks to remove unwanted knowledge or capabilities while preserving other performance, and model editing seeks precise changes with desirable generalization and limited effects on other knowledge.

• The review reports moderate steering successes, but notes that unlearning methods modifying intermediate activations without changing weights had not produced competitive results in the cited evaluation.
• Overlapping mechanisms make precise edits difficult, and cited critiques question whether existing localization identifies the right components to edit.
• Studies of finetuning, additional training of an existing model, suggest that some changes modify existing circuitry shallowly and that refusal training can be reversed. These findings motivate comparing model internals across versions and testing whether capabilities were removed or merely masked.

3.3 Using mechanistic interpretability for better predictions about AI systems

Mechanistic analysis could help predict behavior in unfamiliar situations by identifying failure-related mechanisms or internal indicators of factual knowledge and hallucination, meaning factually unsupported model output. The paper distinguishes these partial predictive successes from establishing the absence of dangerous mechanisms.

• Formal verification would establish that a specified property holds for every input in a defined distribution.
• The cited evidence includes accuracy bounds for a single-layer transformer on a synthetic task and conversion of simple trained networks into programs.
• Scaling such results to large systems remains unresolved, especially when behavior depends on complex environments that may resist formal specification.

Predicting capability development requires understanding how mechanisms change during training and finetuning. The review connects evidence of stagewise learning, changes in parameter trajectories on synthetic data, and correlations between emerging attention circuits and in-context learning, the use of examples supplied in the current input.

• These observations motivate further research; they do not establish a general capability forecasting method. The review also notes disagreement over how to interpret apparently abrupt capability emergence with scale.
• Training data composition and the distinction between existing, elicitable, and masked capabilities are further open problems.
• Many capabilities involve sequential behavior, while current analysis predominantly examines predictions at a single time step.

3.4 Using mechanistic interpretability to improve our ability to perform inference, improve training, and make better use of learned mechanisms

Understanding component functions could support faster inference by identifying computations that can be skipped, and improve distillation, the transfer of learned behavior or information between models. Other proposed uses include selecting training data, tracking capability development, and updating only task-relevant parameters.

• The section also discusses correcting reasoning failures, combining model parameters, and transferring representations.
• These proposals are supported by examples from prior work rather than a common evaluation establishing their effectiveness across applications.

3.5 Using mechanistic interpretability for ‘microscope AI’

“Microscope AI” means training a model on data and interpreting what it learns to discover useful, potentially novel predictors. Cited examples include extracting chess concepts from AlphaZero for grandmasters and identifying previously unknown morphological features associated with immune cell protein expression.

• The scientific value depends on interpreting and validating the discovered patterns.
• Access remains limited by the combined need for machine learning, interpretability, and domain expertise.

3.6 Mechanistic interpretability on a broader range of models and model families

The review observes that research has concentrated on Convolutional Neural Networks (CNNs), image models using convolutional operations, and BERT- and GPT-based text models, two families of transformer language models. Whether methods and conclusions transfer to other architectures and modalities remains open.

• The paper cites initial transfers to state space models, sequence models using evolving internal states, while calling for broader comparisons.
• The universality hypothesis proposes that different networks learn similar features and circuits; evaluating it requires distinguishing shared structure from findings specific to an architecture or model.

3.7 Human computer interaction with model internals

Interfaces to model internals could help engineers investigate representations, auditors examine failures, and users calibrate trust. Proposed conversational dashboards would display influential internal features during interaction and potentially expose safety-related warnings.

• Reliable improvements in user decisions remain to be established for these proposed interfaces.
• A broader design question is how to combine direct manipulation of internal features with ordinary text interaction.

4 Open socio-technical problems in mechanistic interpretability

Practical interpretability raises questions about governance, institutional incentives, and the meaning of successful explanation. Technical findings must be assessed within the social settings in which models are developed and deployed.

4.1 Translating technical progress in mechanistic interpretability into levers for AI policy and governance

Possible governance applications include stronger risk evaluations, capability forecasting, incident investigation, deployment monitoring, and explanations of consequential decisions. Detecting and removing memorized copyrighted works is another proposed use.

• The discussion relates these possibilities to the European Union’s AI Act and General Data Protection Regulation, and to copyright questions as presented in the cited sources.
• The authors acknowledge uncertainty about how technical advances will translate into effective governance.
• Verification of regulatory or treaty compliance is presented as a more speculative application.

4.2 Social and philosophical problems in mechanistic interpretability

The field lacks agreement about its goals, definitions, and success criteria, and the review notes that few interpretability tools are used to make leading systems safer in practice. Engineering claims therefore require comparisons with alternative approaches, while scientific understanding can remain a separate research objective.

• An interpretable method need not strictly outperform an uninterpretable baseline if comparable performance comes with useful interpretability benefits.
• Interpretations of concepts such as fairness depend on development and deployment context.
• Selective disclosure, financial conflicts, and overstated claims can turn partial transparency into misleading reassurance. The review cites lobbying claims that incorrectly portrayed the black-box problem as resolved.

5 Conclusion

The conclusion prioritizes stronger foundations for decomposition, more robust causal validation, and explanations of how mechanisms evolve and interact. It recommends pursuing scientific understanding alongside practical applications, supported by better benchmarks and comparisons with approaches that do not require interpretability.

• Potential governance and safety benefits remain conditional on solving technical problems and avoiding misuse and false assurance.

Author Contributions

Lee Sharkey managed the project, coordinated contributors, and contributed substantially to planning, writing, editing, and synthesis. Bilal Chughtai contributed substantially to framing, writing, and editing and created all figures; the contribution statement credits the listed authors with drafting, editing, and feedback.

• The authors thank Fayth Tan for editing assistance and Schmidt Sciences for feedback and funding.
• The abstract states that Apollo Research synthesized the authors’ perspectives on behalf of Schmidt Sciences and that the text need not represent every individual author’s or institution’s views.

Appendix

• “A Summary of open questions” consolidates the research agenda. Under “A.1 Open problems in mechanistic interpretability methods and foundations”, the reverse-engineering questions concern valid decomposition units, linear representations and superposition, SDL errors and scaling, feature geometry, connections to generalization theory, and interpretable model construction with limited performance cost. The remaining methodological questions ask how to describe unfamiliar concepts without imposing human expectations, measure causal effects efficiently, distinguish causal pathways from compensatory responses, and validate hypotheses beyond researcher intuition using reference networks, benchmarks, and tests across distributions. Concept-based probing, circuit discovery, and automation raise questions about data generation, generalization, arbitrary task coverage, component interactions, and possible sabotage of automated interpretation by misaligned systems. “A.2 Open problems in applications of mechanistic interpretability” asks how to build reliable evaluations and monitors, improve adversarial testing, make steering and editing precise, and assess unlearning and finetuning. Its predictive questions cover unfamiliar inputs, formal verification, absence of dangerous mechanisms, capability emergence, training data effects, and capabilities that prompting or finetuning could reveal; further questions address efficient computation, scientific discovery, transfer across architectures, and usable interfaces. “A.2.8 Governance” asks how to identify failure mechanisms, trace incidents, detect harmful processes, and verify changes or compliance. “A.2.9 Open socio-technical problems in mechanistic interpretability” extends this agenda to capability evaluation and forecasting, incident prevention, identifying workloads run on graphics processing units (GPUs), copyright, the field’s definitions and success criteria, and communication that reduces misuse.

Brief Thoughts

The review offers a useful distinction between internal indicators, plausible component descriptions, and validated explanations of computation. The reported reconstruction-substitution losses and sensitivity of circuit metrics to intervention choices justify testing predictive claims before treating a learned feature or subgraph as a mechanistic explanation. For engineering use, a concrete standard follows: evaluate a specified task against relevant alternatives and test whether the explanation predicts interventions or failures. The paper provides a research agenda rather than a single experimental protocol or an empirically established ranking of priorities. Its proposed applications have different evidence requirements, especially when the goal is to establish the absence of dangerous behavior.