On-device AI versus cloud AI: what runs where, and why it matters
Some AI features work with no connection and others need a data center. The split comes down to size, speed, privacy and cost.
Kivorent News desk, , 2 min read
What to know
- Cloud AI sends your request to large models in a data center. On-device AI runs a smaller model on your own phone or computer.
- On-device is private, works offline and responds quickly. Cloud models are more capable.
- Most products now mix the two and decide per request.
When you ask an assistant a question, the answer is computed in one of two places: on the device in your hand or on a server far away. Which one it is affects what the feature can do, how fast it feels and who sees your data.
Cloud AI
Your request travels over the internet to a data center, where a large model runs on specialized chips, and the answer travels back.
What it is good at: the hardest work. The largest models need far more memory and processing power than any phone or laptop has.
What it costs you: a connection is required, there is a short delay, and your data leaves your device. The company running the service decides how long to keep it and what to do with it, within its stated policy.
On-device AI
A smaller model is stored on your phone or computer and runs on its own chips. Recent devices include a processor built for this work, often called a neural processing unit.
What it is good at: privacy, since nothing leaves the device. It also works on a plane, and it responds without a network round trip.
What it costs you: capability. To fit, models are shrunk, commonly by storing their numbers at lower precision. That trades some quality for a large saving in memory. Running a model also uses battery and storage.
Side by side
| On-device | Cloud | |
|---|---|---|
| Your data | Stays put | Sent to a server |
| Works offline | Yes | No |
| Delay | Minimal | Depends on the network |
| Capability | Good for focused tasks | Best available |
| Who pays for computing | You, in hardware and battery | The provider, passed on in fees |
What typically runs where
On-device models handle tasks that are narrow and frequent: transcribing speech, suggesting the next word, sorting photos, summarizing a notification, translating a sign through the camera.
Cloud models handle work that is open-ended: long research questions, writing and reviewing large amounts of code, generating video.
The hybrid middle
Most assistants now route between the two. A simple request stays local. A harder one goes to the cloud, sometimes after asking you. The details to check in any product are what gets sent, whether you are told when it happens, and whether you can turn it off.
Questions worth asking
- Does this feature work in airplane mode? That is the quickest test of where it runs.
- When data is sent, is it stored, and is it used to train models?
- For work use: does company policy allow this data to leave the building at all?
For developers, the local option keeps getting more practical. See our report on a coding model that runs on your own machine.
Spotted an error? Tell us and we will fix it.