multimodal model

basicwidespread

An artificial intelligence model capable of interpreting not only text but also images, audio, or video, and often responding in multiple formats.

In plain terms

Early chatbots could only read and write: they received text and returned text. A multimodal model is like a colleague to whom you can not only write a letter but also show a photo, play an audio recording, or place a table in front of them, and they understand all of it.

'Modality' refers to the form of information: text, image, audio, video. A multimodal model handles these together. In the same conversation, you can ask about a photo, dictate to it, and in many systems, request a drawing or a read-aloud response.

It is useful because most of the world does not exist as text. An invoice, an X-ray, a machine's noise, or a hand-drawn sketch are all types of information that do not need to be converted into words first.

How it works

The model breaks every input into small pieces: text into word fragments, images into small square patches, and audio into short sound segments.
It converts these pieces into number sequences, ensuring that things with similar meanings receive similar sequences, whether they come from words or images. The word "dog" and a photo of a dog thus end up close to each other.
In many models, the image or audio is first converted by a separate part trained for this purpose. Then the shared neural network processes it together with the text, so the model can connect what it sees with what you ask it.
It produces the answer step by step: mostly as text, but some models can also directly create images or speech.
It learned all this from vast amounts of related examples, such as images with their corresponding descriptions or videos with their audio tracks.

An everyday example

You photograph the contents of your fridge and ask the AI assistant running on your phone what you could cook for dinner. The model recognizes the ingredients in the image and suggests a recipe, even though you entered no ingredients.

Who is affected

everyonePhone assistants can now respond to images, audio, and live camera feeds, so in many situations it is enough to show what you previously had to type.

employeeScanned documents, diagrams, screenshots, and audio recordings of meetings can also be processed with it, not just typed text. → Work & HR

parent, student, teacherIt can explain a photographed math problem or diagram step by step, which helps learning but also makes it easier to have the AI do the homework instead. → Education & Family

developerSuch models underpin agents that manage programs and websites based on screenshots. → Code

A common misconception

MythThe multimodal model sees and hears like a human.

In realityThe model recognizes patterns in images and sounds converted into numbers; it has no experience or real spatial sense. Therefore, it makes mistakes that a human rarely does: it miscounts objects or misreads a clearly visible sign.

Limits and risks

It can confidently err on images too: it may misread numbers, dates, or small text, so always verify important data against the original.
The uploaded photo or audio recording may contain personal data (face, address, medical report), so consider what you share and with whose service.
In interpreting medical images, technical drawings or legal documents, it does not replace a specialist and is at best useful for initial orientation.
Processing images, audio, and video requires more computation than text, making it slower and potentially more expensive, and many models struggle with long videos.

Similar, but not the same

large language model – A large language model originally works only with text. A multimodal model also handles images, audio, or video, and is often an extension of a language model.
image generation model – An image-generating model typically creates images from text descriptions, meaning it specializes in a single task. A multimodal model can also understand images and converse about them. Today, many multimodal models can also generate images.
AI-agent – An agent independently executes tasks in multiple steps. The multimodal model provides the capability for the agent to perceive the screen, images, or audio.

Related concepts

Broader

artificial intelligence, generative AI, foundation model, machine learning

Narrower

vision-language model

On the edge

large language model, image generation model, speech recognition, text-to-speech, Computer vision, embedding, token, AI-agent

A brief history

Joint machine processing of images and text has been researched for a long time. A turning point was brought by OpenAI's CLIP model, introduced in 2021, which learned to connect images and their descriptions in a shared representation space. From 2023, major chat applications also gained image understanding (for example, GPT-4V and Gemini). In 2024, systems appeared that handle text, images, and audio in a single model in real time, such as GPT-4o.

Where things stand

By the end of 2026, leading AI assistants are multimodal by default: they accept images, documents, and audio, and many operate with live camera feeds and speech. Reliable understanding of long videos and precise spatial reasoning remain developing areas.

Status: 2026 November

In the news

Appears in 2 Agent Age stories; the curve shows weekly occurrence in the news magazine.

DeepSeek releases multimodal model and plans Huawei chip order as US–China AI talks nearAug 31, 2026 · Agent Age
Chinese open models close in on the Western frontier: Qwen, GLM-5.3, DeepSeek, MoonshotAug 17, 2026 · Agent Age

Also known as

multimodal model · multimodal AI · multimodal artificial intelligence · multimodal large language model · multimodal model · multimodal AI · multimodal LLM · MLLM

Sources

Radford et al. (2021): Learning Transferable Visual Models From Natural Language Supervision (CLIP)

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.