Research frontier Research

Interpretability (looking inside models)

Research that tries to understand what happens inside AI models and why they answer as they do.

Forward-looking: this describes work in progress, not a finished product. Nothing here is a promised date or release.

What is it?

Modern AI models are made of billions of numbers learned during training. They work well, but even their creators cannot easily explain why a model gives a specific answer. Interpretability research tries to open this 'black box' and find which internal parts represent concepts and drive behaviour.

Researchers have found ways to identify some features inside models, such as patterns linked to specific topics, and to trace parts of how a model reaches an answer. This could help detect hidden problems, improve safety and build trust. The field is young, and current methods explain only a small part of what large models do.

Which models use it?

Anthropic has published research on finding interpretable features inside Claude models.
OpenAI and Google DeepMind have also published interpretability research on language models.
Techniques studied include sparse autoencoders, probing and attribution methods.
Simpler explainability tools, such as SHAP and LIME, are used to explain predictions of traditional machine learning models.

How to use it

Read accessible research blog posts from AI labs that explain interpretability findings.
Remember that when an AI explains its own reasoning, that explanation may not reflect what actually happened inside.
If you code, try explainability tools like SHAP on a simple model to see which inputs matter.
Explore open-source interpretability libraries that work with small open models.
Follow discussions on how interpretability could support AI safety and regulation.

Benefits

Could help find and fix harmful or biased behaviour before deployment.
Supports trust and accountability in high-stakes uses.
Helps scientists understand how learning systems represent knowledge.

Limits and risks

Current methods explain only small parts of very large models.
Findings can be hard to verify and may not generalise between models.
An AI's written explanation of its reasoning is not proof of how it worked internally.
Turning research results into practical safety tools is still early.

Prompt template

Copy it, replace the parts in [brackets] and paste it into any AI chat.

Explain AI interpretability to me as a [beginner / developer / policy reader]. Cover: why models are hard to understand, the difference between interpretability and an AI's own self-explanation, [2] example research methods in plain words, and why it matters for safety. Clearly separate what is known from open questions, and suggest next steps for learning.