AI & TECHNOLOGY — Local Inference

You Can Run a Frontier-Class AI on Your Laptop Now — How the Local AI Revolution Worked

Two years ago, running a frontier artificial-intelligence model meant renting time on a cloud GPU. Today a 64-gigabyte MacBook, an affordable desktop card, or even a phone can run the same kind of model at usable speed, with the data never leaving the device. The change was not a single breakthrough — it was several efficient techniques stacking up at once.

• Frontier open-weight models such as Qwen, Llama, Gemma, and DeepSeek now ship in compressed formats that fit inside ordinary graphics memory and consumer-class RAM.
• Apple Silicon's unified memory lets a 64 GB MacBook run models that would not fit on a 24 GB RTX 4090 desktop card, and Apple's native MLX framework beats desktop runtimes on the chip by 30 to 40 percent.
• A small software layer — Ollama, llama.cpp, vLLM and friends — makes downloading, quantising and running a model a single command for anyone with a modern computer.

The key technique is quantisation. A frontier model as shipped by its creator stores each number in full floating-point precision, which is precise but bulky. By rounding the model's numbers — typically to four or eight bits — the same model shrinks to a quarter or half its original size with only a small hit to quality. A 4-bit version of a model that once required a 24 GB GPU card can now run on 8 GB of laptop memory, fast enough for chat, code assistance and agentic workflows.

Memory architecture matters as much as the model. NVIDIA GPUs separate VRAM from system RAM, so a desktop card is limited by the VRAM slot. Apple Silicon shares a single pool of unified memory across CPU, GPU and Neural Engine, so the entire chip memory is available to the model. On the newest M-class chips, open-weight models routinely produce 60 to 85 tokens per second locally — good enough that most users cannot tell the difference between a local response and a cloud one.

The practical consequence is that privacy, latency, cost and control now favour local deployment for many workloads. A law firm can run legal-reasoning models on its own server without sending client data through an API. A developer can point a coding agent at the full source tree locally. A hospital can run medical models on private infrastructure. None of this requires a monthly subscription or trust in a remote provider's terms of service. Local AI is becoming the default for anyone whose data should not travel.

The hardware limit that remains is bandwidth, not raw capacity. A local model is only as fast as the data can move between memory and the compute cores, which is why high-speed unified memory on Apple Silicon or a wide PCIe interface on a desktop card matters more than headline transistor counts. As models grow and hardware follows, the line between "cloud model" and "laptop model" keeps thinning.

Knowledge takeaway: frontier AI models can now run on ordinary laptops and phones thanks to quantisation that shrinks model size, shared unified memory on Apple Silicon, and simple open-source runtimes; local deployment now wins on privacy, cost, latency and control for many real-world workloads; the limiting factor on modern devices is memory bandwidth, not raw compute or storage capacity.