on-device AI

basicwidespread

Artificial intelligence that works directly on your phone, laptop, or other device, without sending your data to a remote server.

In plain terms

Most AI services work by sending your query over the internet to a massive server farm, where the answer is generated and sent back to you. With on-device AI, there is no travel: the model lives on your device and computes locally.

It is like the difference between a pocket calculator and phone directory assistance. The calculator is always at hand, answers instantly, and no one hears what you asked. The directory assistance service knows more, but it needs a line, you have to wait for it, and someone on the other end learns what you are curious about.

There are three advantages over cloud solutions: your data does not need to leave the device, it works without internet, and there is no network latency. The trade-off is that only a smaller, less capable model fits in a phone compared to a data center.

In practice, the two often blend: the device handles simpler tasks locally and forwards harder ones to the cloud. This is called hybrid operation.

How it works

Developers first train a model on large machines, then shrink it: they store numbers with lower precision (quantization) or distill the knowledge of a large model into a smaller one (distillation).
The downsized model is installed on your device, typically as part of an app or system update, and occupies storage space, sometimes several gigabytes.
When you use it, the computation is performed by the device's own chip. Newer phones and laptops have a dedicated component for this, the neural processing unit (NPU), which runs AI operations quickly with low power consumption.
The input (your voice, photo, text) and the result remain on the device unless the application explicitly sends them further.
If a task is too large for the local model, the system can turn to the cloud in hybrid mode. A well-designed application will indicate this to you or request permission.

An everyday example

You are sitting on a plane with airplane mode on, yet your phone still recognizes your face to unlock, on many devices finds all the "dog" photos in your gallery, and, in supported languages, transcribes your dictated notes. All of this works without the internet, because the work is done by a model running on your device.

Who is affected

everyoneYour phone's face recognition, dictation, photo organization, and most typing suggestions already run on the device today.

entrepreneur, executiveFor sensitive business data, a key consideration is that processing takes place on the corporate machine, not on an external provider's server.

patient, doctorHealth data, such as smartwatch heart rate analysis, can be evaluated locally, so less sensitive data leaves the device. → Health

developerIt requires a different design: it must cope with limited memory, battery and many kinds of hardware, but in return it pays no server costs for each request. → Code

A common misconception

MythIf a phone or laptop is sold as an 'AI device', all its AI functions run locally, and no data leaves the device.

In realityMost such devices are hybrid: certain functions run locally, others in the cloud. You can determine which is which only from the settings and the data processing notice.

Limits and risks

A model that fits on the device is smaller, so it is weaker on complex questions and makes mistakes more often than large cloud models.
Local execution drains the battery, heats the device, and consumes storage; on older devices, many features are simply unavailable.
Local processing alone is not a guarantee of data protection: the application may still forward the result or usage data.
The model's knowledge only expands with updates, and without internet access, it cannot access fresh information.

Similar, but not the same

cloud-based AI – There, the computation happens on the provider's remote servers; your device only sends the request and displays the answer.
Edge computing – Broader concept: the computation takes place near where the data originates, but this can be a nearby local server or network device, not just your own device.
helyi nyelvi modell (local LLM) – This is a type of on-device AI where you run a text-based language model specifically on your own machine; on-device AI is broader than this, also including image and speech recognition.

Related concepts

Broader

edge computing, AI infrastructure, inference

Narrower

helyi nyelvi modell, small language model, TinyML, AI PC

On the edge

neural processing unit, quantization, model distillation, federated learning, cloud-based AI

A brief history

The turning point came in 2017: Apple introduced the A11 Bionic chip with an integrated Neural Engine, and Google launched the TensorFlow Lite framework for mobile (renamed LiteRT in 2024). The shift of generative AI to devices gained major momentum in 2024 with the launch of Copilot+ PCs and Apple Intelligence.

Where things stand

Most facial recognition and photo processing on new phones runs locally, as does dictation in supported languages. Most new flagship phones and better laptops now come with an NPU. However, generative features are still largely hybrid at the end of 2026, with heavier tasks continuing to go to the cloud.

Status: 2026 November

In the news

Appears in 6 Agent Age stories; the curve shows weekly occurrence in the news magazine.

JFrog launches local AI security for developer laptops to detect shadow AIOct 21, 2026 · Code
VS Code 1.143 makes Copilot the default, adds AI model bake-off and local AI without creditsOct 19, 2026 · Code
Local AI coding: JetBrains Mellum 2.1, Junie on RTX Spark, OpenCode with OllamaOct 12, 2026 · Code
Is the one-million-forint Ghost Core local AI computer for developers worth the price?Oct 12, 2026 · Code
WhatsApp to screen suspected scam messages with on-device AIAug 31, 2026 · Money
Industrial edge AI: Emerson PACEdge 3.0, LG–Nvidia AI factory and Korean manufacturing AI with 23 billion wonAug 27, 2026 · Ipar

Also known as

on-device AI · on-device AI · on-device AI · locally running AI · local AI · local AI · edge AI · edge AI

Sources

Wikipedia: Federated learning

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.