Research frontier Research

Efficient small models

Making AI models smaller, cheaper and faster while keeping them useful.

Forward-looking: this describes work in progress, not a finished product. Nothing here is a promised date or release.

What is it?

Large AI models need powerful and expensive computers, a lot of energy and often an internet connection. Researchers are working on smaller, more efficient models that can run faster, cost less and even work on phones or laptops.

Common techniques include distillation, where a small model learns from a larger one; quantization, which stores model numbers with less precision; pruning, which removes less important parts; and better training data. Small models have improved a lot, but they still usually trail the largest models on difficult tasks, and finding the best trade-offs is an active area of research.

Which models use it?

Small open model families include Gemma from Google, Phi from Microsoft, and smaller Llama, Mistral and Qwen models.
Major providers offer lighter, faster versions of their models, such as smaller Claude, GPT and Gemini tiers.
Tools like Ollama, LM Studio and llama.cpp run quantized models on personal computers.
Distillation and quantization are widely used techniques in research and industry.

How to use it

For simple tasks like classification, short summaries or rewriting, try a smaller, faster model first.
Compare the output of a small and a large model on your own examples.
If you have a capable computer, run a small open model locally with Ollama or LM Studio.
Choose the smallest model that meets your quality needs to save cost and time.
Follow research on distillation and quantization to see how the gap is changing.

Benefits

Lower cost and faster responses.
Can run on-device for better privacy and offline use.
Uses less energy than running very large models.

Limits and risks

Small models usually make more mistakes on complex reasoning and specialist topics.
Heavy compression can reduce quality in ways that are hard to notice at first.
Running models locally still needs enough memory and processing power.
How far small models can close the gap with large ones is uncertain.

Prompt template

Copy it, replace the parts in [brackets] and paste it into any AI chat.

Help me pick a model size for this task. Task: [describe]. Volume: [requests per day]. Quality needs: [low/medium/high]. Privacy: [must stay on device? yes/no]. Hardware: [if running locally]. Recommend whether a small, medium or large model fits best, explain the trade-offs in cost, speed and accuracy, and suggest a simple test I can run to compare them.