VLM

in English: vision-language model

advancedwidespread

An AI model that understands both images and text simultaneously, so it can say what is visible in a photo and answer questions about it.

In plain terms

A traditional language model only reads text: like a highly educated person talking to you on the phone who cannot see what you see. A vision-language model (VLM) is the "sighted" version: you can show it a photo, screenshot, or scanned invoice and ask questions about it verbally.

It is good for describing what is in an image, reading text on it, interpreting a chart or table, comparing two photos, or identifying what might be wrong with a machine shown in the image. The answer is typically text, not a new image.

Older image recognition programs could only attach pre-defined labels to an image, such as "dog" or "car". In contrast, a VLM answers freely formulated questions and can connect the image with its background knowledge.

Most of today's known chat AIs are already built on such a model. When you upload a photo to a chatbot, you are actually talking to a VLM.

How it works

The model cuts the image into tiny squares, called patches, similar to how text is broken down into words and word fragments.
An image-processing part (image encoder) converts every patch into a number sequence that describes what is visible on that piece: edge, color, letter, facial detail.
In many models, a connecting layer 'translates' these number sequences into the language model's internal language, allowing the language model to handle image fragments similarly to words.
The language model reads elements from the image and your question together, then writes the answer piece by piece, word by word.
The model learned all this on countless image–text pairs (for example, photos and their corresponding captions), forming an understanding of which visual corresponds to which words.

An everyday example

You photograph the error code flashing on your washing machine's display with your phone and ask the chatbot what is wrong. The model reads the code if visible in the image, recognizes the manufacturer, and explains what you should check before calling a repairman. It is worth comparing the answer with the machine's manual.

Who is affected

everyoneIf you show a photo, screenshot, or document to an AI assistant, a vision-language model works in the background.

employeeThe processing of invoices, forms, scanned contracts, and diagrams is increasingly handled by such models, transforming administrative work. → Work & HR

patient, doctorIt also appears in the preliminary interpretation of X-rays, skin lesions, and findings, but the decision and responsibility remain with the doctor. → Health

developerAgents that “see” the screen and document-processing systems typically rely on VLMs, and they are increasingly used in robotics as well. → Code

A common misconception

MythThe vision-language model sees as humans do, so what it says about an image is certainly present there.

In realityThe model converts the image into numbers and formulates an answer based on probabilities. Therefore, it can confidently describe a detail not present in the image or misread a number.

Limits and risks

It may err in small details: miscount objects, confuse left and right, or misread small text or numbers.
It may hallucinate, meaning it can state plausible but fabricated details about an image, so always verify financial, medical, or legal content against the original.
Uploaded photos may contain personal data (face, address, license plate, documents), so consider what you share with a service.
It can be deceived by text instructions hidden in an image, which is especially risky if the model is part of an independently acting agent.

Similar, but not the same

Image generation model – An image generation model creates new images from text, while a VLM interprets existing images and responds with text.
Nagy nyelvi modell (LLM) – A classic LLM works only with text, whereas a VLM can also accept images.
Optical character recognition (OCR) – OCR simply reads letters from an image, while a VLM interprets text and visuals together and can answer questions about them.

Related concepts

Broader

multimodal model, Foundation model, Generative AI

Narrower

Visual Question Answering, Image captioning

On the edge

Nagy nyelvi modell (LLM), Computer Vision, Optical character recognition (OCR), Image generation model, Computer-using agent, Vision-language-action model (VLA)

A brief history

One important precursor was CLIP, introduced in 2021 by OpenAI, which learned from pairing images with their corresponding text. This formed the basis for today's models that answer questions with text, such as DeepMind's Flamingo from 2022. The general public encountered VLMs en masse in 2023, when chatbots like ChatGPT and Bard began accepting images.

Where things stand

At the end of 2026, leading chat AI models understand images by default, and open versions that can run on your own machine are also available. The focus of development is shifting toward video understanding, screen-controlling agents, and robotics.

Status: 2026 November

In the news

Appears in 1 Agent Age stories; the curve shows weekly occurrence in the news magazine.

AI-based HR assistant in hospitality: VLM pre-screens hundreds of CVsAug 31, 2026 · Work & HR

Also known as

VLM · vision-language model · vision language model · visual language model · vision-language model · vision-language model · vision-language model · multimodal language model

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.