Transformers and attention
The neural network design that most modern language, image and speech models are built on.
What is it?
A transformer is a type of neural network. Its key idea is attention: when the model processes a word, it looks at all the other words in the input and decides which ones matter most for understanding it. In "The cat sat on the mat because it was tired", attention helps the model link "it" to "the cat".
Transformers process many pieces of input in parallel, which made it practical to train very large models on huge datasets. The design was first described for translation and is now used for text, images, audio and video.
Which models use it?
- GPT, Claude, Gemini, Llama and Mistral language models are transformer-based.
- BERT, a widely used Google model for search and text classification, is a transformer.
- Vision transformers apply the same idea to images by splitting them into small patches.
- Whisper, OpenAI's speech recognition model, uses a transformer.
- Many image and video generators use transformer components alongside other techniques.
How to use it
- Remember that the model reads your whole input at once, so where you place information matters less than how clearly you label it.
- Put related information together and use headings or labels so the model can connect the right parts.
- Refer to things by clear names rather than vague words like "it" or "that one" when the input is long.
- Keep long inputs focused: unrelated text competes for the model's attention.
- If you are learning the technology itself, start with an illustrated explanation of attention before reading the math.
Benefits
- Explains why modern models handle long, complex inputs well.
- One architecture works across text, images, audio and code.
- Trains efficiently on modern hardware, which allowed today's large models.
- Understanding it helps you write clearer prompts and read AI news critically.
Limits and risks
- Attention becomes expensive as inputs get longer, which is one reason context windows have limits.
- Large transformers need a lot of computing power and energy to train and run.
- How a transformer reaches a specific answer is still hard to explain in detail.
- Knowing the architecture does not tell you whether a given model's output is correct.
Prompt template
Copy it, replace the parts in [brackets] and paste it into any AI chat.
I want to understand how attention works in transformer models. Explain it in [number] short steps using the example sentence "[your sentence]". For each step, say what the model is doing in plain words. Then give me one analogy from everyday life and one limitation of attention. Avoid math unless I ask for it.
More in Foundations
- Large language models (LLMs)The text-predicting systems behind chat assistants: what they are and what they can do.
- Tokens and context windowsHow models measure text, and why there is a limit to how much they can read at once.
- Training vs inferenceThe difference between teaching a model and using it, and why it matters to you.
- Open-weight vs closed modelsModels you can download and run yourself vs models you use only through a company's service.
- Choosing the right model (cost, speed, quality)How to match a model to your task instead of always using the biggest one.