Multimodal AI and vision explained simply

What is it?

A multimodal model can take in more than one kind of input, such as text and images together. You can show it a photo, a chart, a screenshot or a scanned page and ask questions about it in plain words.

Under the hood, the image is turned into numbers the model can process alongside your text. This lets it describe scenes, read printed text, explain diagrams and compare pictures. It does not see the way people do: it can miss small details, misread handwriting or be confidently wrong about what is in a picture.

Which models use it?

GPT models from OpenAI accept images as input in ChatGPT and through the API.
Claude models from Anthropic can read images, charts and PDF pages.
Gemini models from Google are built to handle text, images, audio and video.
Some open-weight model families, such as certain Llama and Qwen variants, offer vision versions.
Phone camera features like Google Lens use computer vision to recognise objects and text.

How to use it

Choose a clear, well-lit image or a sharp screenshot; crop out what does not matter.
Say exactly what you want: describe, extract text, compare, or find a specific detail.
Give context the image cannot show, such as what the chart is about or what you are trying to decide.
Ask for the answer in a useful format, for example a table or a short list.
Check important details (numbers, names, small print) against the original image yourself.

Benefits

Turns photos and screenshots into searchable, editable text.
Explains charts, diagrams and interfaces in plain language.
Helps people with low vision get descriptions of images.
Saves time on simple visual tasks like sorting or tagging pictures.

Limits and risks

Can misread small text, handwriting, tables or numbers in images.
May describe things that are not in the picture (hallucination).
Not reliable for judging medical images, safety issues or identity; use qualified experts.
Images can contain personal data; avoid uploading other people's private information without consent.

Prompt template

Copy it, replace the parts in [brackets] and paste it into any AI chat.

Look at the attached image of [what it is, e.g. a sales chart / a product label / an app screen].

1. Describe what it shows in 2-3 sentences.
2. Extract [the specific information you need, e.g. all numbers in the table] into a table.
3. Point out anything that is unclear, cut off or hard to read instead of guessing.

Context: [why you need this and who will use it].

Multimodal AI and vision