Training vs inference
The difference between teaching a model and using it, and why it matters to you.
What is it?
Training is the phase where a model learns. It is shown huge amounts of data and its internal settings (called parameters or weights) are adjusted again and again until it gets good at its task. For large models this takes a lot of specialized hardware and is done by the model's developer, not by users.
Inference is using a trained model to produce an answer. Every time you send a message to a chat assistant, the model runs inference. It does not normally learn from your message during inference; its weights stay the same. Some products store your chats and may use them to train future models, depending on your settings.
Which models use it?
- Developers such as OpenAI, Anthropic, Google and Meta train their models before release.
- Chat assistants such as ChatGPT, Claude and Gemini run inference each time you send a message.
- Open-weight models such as Llama and Mistral can be run for inference on your own hardware.
- Cloud platforms rent out hardware for both training and inference.
How to use it
- Remember that the model's built-in knowledge comes from training and has a cutoff point.
- Give current or private information directly in your prompt, because inference does not update the model.
- Check your provider's data settings to see whether your chats may be used for future training.
- Turn off training on your data if the setting exists and privacy matters to you.
- For repeated specialized tasks, consider whether better prompts are enough before thinking about training (fine-tuning).
Benefits
- Clarifies why a model does not remember facts you told it in a different chat.
- Explains why models can be unaware of recent events.
- Helps you make informed choices about privacy settings.
- Separates the very high one-time cost of training from the per-use cost of inference.
Limits and risks
- Training data can contain errors and biases that carry into the model.
- A model's knowledge becomes outdated as the world changes.
- Inference on large models still uses significant energy and computing power.
- Memory features in some apps can blur the line, so read how each product stores information.
Prompt template
Copy it, replace the parts in [brackets] and paste it into any AI chat.
I am using an AI assistant for [task]. Tell me which information you might be missing because of your training cutoff, and list the facts I should give you directly so your answer is accurate. Then ask me up to [number] questions to collect that information before you answer.
More in Foundations
- Large language models (LLMs)The text-predicting systems behind chat assistants: what they are and what they can do.
- Transformers and attentionThe neural network design that most modern language, image and speech models are built on.
- Tokens and context windowsHow models measure text, and why there is a limit to how much they can read at once.
- Open-weight vs closed modelsModels you can download and run yourself vs models you use only through a company's service.
- Choosing the right model (cost, speed, quality)How to match a model to your task instead of always using the biggest one.