Apple’s on-device language model, the one that powers writing tools and summaries on a recent iPhone, has about three billion parameters. The frontier models you talk to in a browser are widely believed to be hundreds of times larger. And yet for the jobs Apple actually asks it to do — tidying up a sentence, pulling a date out of an email, ranking notifications — the small one is good enough, runs in the palm of your hand, and costs nothing to call.
That gap between “impressive on a benchmark” and “good enough for this task” is the whole story of on-device AI in 2026. A lot of the work people want from a language model is narrow, repetitive and low-stakes. Narrow, repetitive and low-stakes is exactly what a small model is good at, and running it locally removes the network round trip, the per-token bill, and the awkward question of whose servers your data just visited.
The hardware caught up too. Every major phone and laptop chip shipping now includes a neural processing unit, and the software stacks to use them finally exist. Here’s what’s real, what’s marketing, and how to try it yourself.
What “small” actually means now
There is no formal definition. In practice, a small language model (SLM) is one you can run on a single consumer device — a phone, a laptop, a single mid-range GPU — rather than a rack of accelerators. That puts the working range roughly between 250 million and 15 billion parameters, with 1B to 8B being the sweet spot for laptops and 1B to 4B for phones.
“Parameters” just means the learned numbers inside the model. More of them generally means more stored knowledge and better reasoning, but also more memory and more arithmetic per word generated. Memory is usually the binding constraint on a phone, not raw compute.
The case for going small is not purely economic. A 2025 NVIDIA Research position paper by Peter Belcak and colleagues argues that most AI agents perform a handful of specialised tasks over and over with little variation, which makes a general-purpose frontier model the wrong tool — capable, but wildly oversized for the job. Their recommendation is heterogeneous systems: small models for the repetitive majority, a large one called in only when genuinely open-ended reasoning is needed.
How a big model becomes a small one
Three techniques do most of the work, and they stack.
Distillation
The idea dates to a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, “Distilling the Knowledge in a Neural Network”. A large “teacher” model generates outputs, and a much smaller “student” model is trained to reproduce not just the teacher’s final answers but its full probability distribution over possible answers. That distribution carries more information than a single correct label — it encodes which wrong answers the teacher considered plausible — so the student learns faster and generalises better than it would from raw data alone. Nearly every “mini” or “flash” model on the market today is distilled from a larger sibling.
Quantization
Model weights are normally stored as 16-bit floating-point numbers. Quantization stores them with fewer bits — 8, 5, 4, sometimes fewer — which cuts file size and memory bandwidth roughly proportionally. Bandwidth matters enormously, because generating each token requires reading the model’s weights out of memory.
The obvious question is what you lose. A January 2026 evaluation of llama.cpp quantization on Llama-3.1-8B-Instruct put numbers on it. The full-precision model occupies about 14.96 GiB and scores 69.47% averaged across benchmarks. A 4-bit-ish Q4_K_S version drops to roughly 4.36 GiB — about a 71% size reduction — while scoring 69.17%, a loss of well under half a percentage point. Push down to Q3_K_S at around 3.4 bits and the picture changes: accuracy falls to 65.49% and perplexity rises 22%.
The degradation is also uneven by task. In that study, grade-school maths problems (GSM8K) were the most sensitive to aggressive quantization, while commonsense reasoning stayed comparatively stable. If your use case is arithmetic or code, be more conservative than if it is summarising email.
The alternative to squashing a finished model is training it to be squashed. Apple uses quantization-aware training, where the compression is simulated during training so the model learns weights that survive it.
Pruning and sparsity
The newest trick is to keep a large model on disk but only load part of it. Apple’s third-generation on-device models, announced on 8 June 2026, include AFM 3 Core Advanced: 20 billion parameters total, but structured so that only 1 to 4 billion are active for any given request. The full model lives in flash storage rather than RAM, and routing decisions made per prompt pull the relevant “expert” sections into memory. Apple calls the underlying technique Instruction-Following Pruning.
Google took a related approach with Gemma 3n. Its E2B variant contains over 5 billion parameters on paper but has an effective memory load of about 1.91 billion, achieved by caching per-layer embeddings to fast storage and streaming them in during inference. The “E” in the name stands for effective parameters, which is a genuinely useful distinction and one worth watching for in spec sheets.
The silicon in your pocket and on your desk
NPUs are matrix-multiplication accelerators tuned for low-precision integer maths at very low power. They are usually slower than the GPU in the same chip at peak throughput, but far more efficient per operation, which is what matters when you are running on a battery.
Phones
Qualcomm’s Snapdragon 8 Elite Gen 5 is the current Android flagship part. Per Counterpoint Research, it is built on TSMC’s N3P process, pairs two prime CPU cores at up to 4.6GHz with six performance cores at 3.6GHz, and its Hexagon NPU is claimed to be 37% faster than the prior generation. More interesting than the speed claim is the addition of INT2 mixed-precision support — the chip can natively handle weights quantized down to two bits. Devices using it began shipping in volume in early 2026, including the Galaxy S26 line.
Google went a different route, designing its own TPU block into the Tensor G5. In July 2026 it announced Gemma 4 E2B for TPU, a variant compiled to run natively on that block across the Pixel 10 family, and the basis for Gemini Nano 4. The pitch is offline multimodal work: image recognition and audio transcription with no network at all.
Laptops
Microsoft drew a hard line with Copilot+ PCs. Per the Copilot+ PC developer documentation, the badge requires an NPU capable of 40 or more TOPS (trillion operations per second). Qualifying silicon includes Qualcomm’s Snapdragon X Elite, Intel’s Core Ultra 200V series, and AMD’s Ryzen AI 300 series. TOPS is a crude metric — it says nothing about memory bandwidth or what precision the number was measured at — but it did force a floor across the Windows ecosystem.
Apple splits the work between a 16-core Neural Engine and neural accelerators built into each GPU core. The M5, announced in October 2025, offers 153GB/s of unified memory bandwidth, up nearly 30% on M4. The M5 Pro and M5 Max that followed in March 2026 reach 307GB/s with up to 64GB, and 614GB/s with up to 128GB, respectively. For local inference, that bandwidth figure predicts tokens-per-second far better than any TOPS number does.
What actually runs locally today
| Model | Sizes | Where it runs | Notes |
|---|---|---|---|
| Apple AFM 3 Core | 3B dense | Recent iPhone, iPad, Mac | Exposed to apps via the Foundation Models framework |
| Apple AFM 3 Core Advanced | 20B sparse, 1–4B active | Recent Apple silicon | Weights held in flash, experts loaded per prompt |
| Gemma 3n | E2B / E4B | Android, desktop | Text, image and audio in; 32K context |
| Gemma 4 E2B for TPU | E2B | Pixel 10 family | Compiled for the Tensor G5 TPU; underpins Gemini Nano 4 |
| Phi-4-mini | 3.8B | Any laptop with ~8GB free RAM | Reasoning and function calling |
| Llama 3.2 | 1B / 3B | Phones and laptops | Widely used baseline for local apps |
| Qwen2.5 | 0.5B upward | Phones and laptops | Strong multilingual coverage |
| SmolLM2 | 135M / 360M / 1.7B | Very constrained devices | Useful for classification and extraction |
Two platform notes are worth knowing. Apple opened its on-device model to third-party developers with the Foundation Models framework in September 2025, with inference free of charge and working offline — a meaningful change to app economics, since developers no longer pay per token for basic language features.
On Windows, Microsoft’s built-in local model, Phi Silica, runs on Copilot+ NPUs and also on NVIDIA GeForce RTX 30-series or newer and AMD Radeon RX 9060-series or newer GPUs with at least 6GB of video memory. Microsoft’s own Phi Silica documentation notes that it is being replaced by a successor called Aion Instruct, with rollout beginning October 2026 and Phi Silica removed in November 2026. If you are building on it, plan for that.
Privacy and latency, honestly assessed
The privacy argument is the strongest one and the simplest. If inference happens on the device, the text never leaves it. For a Canadian clinic handling patient notes, a law firm with privilege obligations, or anyone under a data residency requirement, that is not a nice-to-have — it is often the only way the feature is permissible at all. It also removes an entire category of vendor risk: no retention policy to audit, no subprocessor list, no breach at a third party exposing your prompts.
Latency is the second benefit, and it is more nuanced than “local is faster.” A small local model usually beats a cloud model on time-to-first-token, because there is no network round trip. On sustained generation speed, a well-provisioned cloud model often wins. Where local really shines is predictability: no rate limits, no queueing behind other customers, no outage. Offline operation on a plane or in a basement is a genuine capability difference, not a marginal one.
Cost is the third. Running on hardware the user already bought is free at the margin, which changes what features are worth shipping: you can afford to run a model on every keystroke.
Where local models still fall down
Be clear-eyed about the trade-offs.
- World knowledge is thin. A 3B model simply cannot store what a much larger one does. Ask it about a niche historical event or an obscure API and it will often produce something confident and wrong.
- Long, multi-step reasoning degrades. Chains of inference where each step depends on the last are where the size gap is most visible.
- Hallucination is not a size problem you can fix by going bigger. A Nature paper published in April 2026 by Adam Tauman Kalai and colleagues, “Evaluating large language models for accuracy incentivizes hallucinations”, argues the root cause is that standard accuracy-based evaluations reward confident guessing over admitting uncertainty. That incentive applies at every scale.
- Context windows are shorter. Gemma 3n offers 32K tokens; frontier cloud models offer far more. Long-document work is still a cloud job.
- Thermals and battery. Sustained generation on a phone throttles. Fine for a paragraph, painful for a long report.
- Fragmentation. Every vendor’s NPU has its own toolchain. A model tuned for one runs poorly, or not at all, on another.
What this means for you
If you are choosing hardware, ignore TOPS marketing and look at two things: total RAM, and memory bandwidth. A model must fit in memory to run at all, and bandwidth sets how fast it generates. 16GB is a realistic floor for comfortable local inference on a laptop in 2026; 32GB gives you room for larger models plus your actual work.
If you want to try it, the shortest path is Ollama, which is free, runs on macOS, Windows and Linux, and handles downloading and quantization for you. Install it, open a terminal, and run ollama run gemma4. The first launch downloads the model; after that it works offline. Swap in llama3.2:3b or phi4-mini to compare. Start with a 3B or 4B model at 4-bit quantization — that is the configuration most likely to feel usable on ordinary hardware.
A useful rule of thumb: budget roughly the model’s file size in gigabytes plus two to three more for the operating system and the conversation. A 4.5GB quantized 8B model wants about 8GB free.
Then be deliberate about routing. Send classification, extraction, rewriting, tagging and summarising of short text to the local model. Send research, long documents, hard code and anything where being wrong is expensive to a large model. Most production systems in 2026 do exactly this, and the routing logic is where the engineering effort now goes.
Frequently asked questions
Do I need a new computer to run a language model locally?
No. Small models in the 1B to 4B range run on ordinary CPUs with 8GB of RAM, just slowly. An NPU or a discrete GPU makes it faster and much easier on the battery, but it is not a hard requirement.
Is a quantized model meaningfully worse?
At 4 to 5 bits per weight, usually not enough to notice for everyday tasks — the January 2026 llama.cpp study found accuracy losses under half a percentage point at Q4_K_S. Below about 4 bits, quality falls off noticeably, especially on maths and code.
Does running locally mean my data is definitely private?
It means the model inference is local. The surrounding app may still send telemetry, sync your files, or fall back to a cloud model when the local one struggles. Check what the specific app does rather than assuming.
What is the difference between a small model and a “distilled” one?
Small describes size; distilled describes how it was trained. Most well-regarded small models are distilled from a larger teacher, but you can also train a small model from scratch on carefully curated data, which is the approach Microsoft’s Phi family is known for.
Will small models eventually match frontier models?
On narrow, well-defined tasks they already do. On open-ended reasoning and breadth of knowledge, the gap has narrowed but has not closed, and the frontier keeps moving. Assume the gap persists and design around it.
The right tool for the size of the job
The real shift is not that small models got smart enough to replace big ones. It is that the industry stopped assuming one model should do everything. A 20-billion-parameter model that keeps its weights in flash and activates two billion at a time, a 5-billion-parameter model that behaves like a 2-billion-parameter one in memory, a 4-bit quantization that costs half a percentage point for a 71% size reduction: these are answers to the question of how much model a task actually needs.
For most people the consequence is invisible and good — features that work without a connection and never send your text anywhere. For anyone building software, it is a decision to make task by task. Routing everything to the largest available model is no longer the obvious default, and in 2026 it is often the wrong one.
Sources
- Apple Machine Learning Research — Introducing the Third Generation of Apple’s Foundation Models
- Apple Newsroom — Apple’s Foundation Models framework unlocks new intelligent app experiences
- Apple Newsroom — Apple unleashes M5, the next big leap in AI performance for Apple silicon
- Apple Newsroom — Apple debuts M5 Pro and M5 Max
- Microsoft Learn — Copilot+ PCs developer guide
- Microsoft Learn — Get started with Phi Silica in the Windows App SDK
- Google AI for Developers — Gemma 3n model overview
- 9to5Google — Google announces Gemma 4 optimized for the Pixel 10’s TPU
- Counterpoint Research — Qualcomm Snapdragon 8 Elite Gen 5
- arXiv — Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
- arXiv — Small Language Models are the Future of Agentic AI
- arXiv — Distilling the Knowledge in a Neural Network
- Nature — Evaluating large language models for accuracy incentivizes hallucinations
- Ollama — Model library
Image credit: Photo: TorQue Astur — CC BY-SA 4.0 (via Wikimedia Commons)
