Multimodal reasoning
Research on AI that reasons across text, images, audio and video together.
Forward-looking: this describes work in progress, not a finished product. Nothing here is a promised date or release.
What is it?
Many AI models can now take in more than one type of input, such as text and images, and some can handle audio and video too. Multimodal reasoning goes a step further: combining information from these sources to solve problems, for example reading a chart and a paragraph together, following a diagram, or understanding what happens over time in a video.
Models can do this well in many simple cases, but research shows they still make mistakes with spatial relationships, counting, fine details, long videos and multi-step problems that mix images and text. Researchers are working on better training, evaluation and reasoning methods.
Which models use it?
- Models such as the GPT, Claude and Gemini families accept images alongside text, and some also accept audio or video.
- Open multimodal models, such as LLaVA and some Llama and Qwen variants, are used widely in research.
- Researchers use benchmarks with charts, diagrams, science questions and videos to measure multimodal reasoning.
- Robotics and self-driving research combine vision and language for decision-making.
How to use it
- Upload an image, chart or screenshot to a multimodal assistant and ask specific questions about it.
- Ask the model to describe what it sees first, then reason step by step.
- Check details like numbers, labels and positions yourself, since these are common error points.
- Try the same question with a clearer or cropped image to see how results change.
- Follow research summaries to learn which tasks remain difficult.
Benefits
- Lets AI help with real-world material such as photos, documents, charts and diagrams.
- Makes AI more accessible, for example describing images for people with low vision.
- Could support science, education and engineering tasks that mix visuals and text.
Limits and risks
- Models can misread small text, count objects wrongly or confuse positions.
- They may describe things that are not in the image.
- Understanding long videos and complex diagrams is still difficult.
- Strong results on tests do not always carry over to real-world use.
Prompt template
Copy it, replace the parts in [brackets] and paste it into any AI chat.
Look at the [image / chart / diagram] I am sharing. Step 1: describe exactly what you see, including labels and numbers. Step 2: answer my question: [your question]. Step 3: list any parts you could not read clearly or are unsure about. Do not guess values that are not visible.
More in Research frontier
- World modelsResearch on AI that builds an internal model of how the world works. Early and uncertain.
- Long-term memory for AIHow AI could remember useful information across conversations over time. An active research area.
- Continual learningResearch on AI that keeps learning new things without forgetting old ones.
- Interpretability (looking inside models)Research that tries to understand what happens inside AI models and why they answer as they do.
- Efficient small modelsMaking AI models smaller, cheaper and faster while keeping them useful.
- Alignment researchResearch on making AI systems reliably follow human intentions and values.