LLMs Locally With a CPU? I Tested 8 Models on Linux
Recent advancements in model formats and quantization techniques have made it possible to run large language models (LLMs) on CPU-only setups. The key metric for usability is the tokens per second rate, which determines how responsive a model feels during use. This article provides insights into which models are suitable for low-end machines based on practical testing.
- ▪Newer model formats and aggressive quantization have made LLMs smaller and lighter, allowing them to run on CPUs.
- ▪The usability of a model is determined by its tokens per second rate, with 15–30 tok/s being optimal for everyday use.
- ▪Models in the 1B-2B range consistently offer a good balance of performance and resource usage on typical older laptops.
Hacker News (Newest) files mainly under programming. We currently carry 5,270 of its stories.
Opening excerpt (first ~120 words) tap to expand
For the longest time, I assumed running LLMs locally needed a decent GPU. That’s what most guides implied, and honestly, that’s how the ecosystem felt not too long ago. But after digging into recent tools and actually trying things out on CPU-only setups, that assumption doesn’t really hold anymore.Newer model formats like GGUF and aggressive quantization (think 4-bit variants) have made these models much smaller and lighter. At the same time, runtimes such as Llama.cpp have become efficient enough that CPUs (yes, even older ones) can run them without completely falling apart.That said, I quickly realized something more important: just because a model runs doesn’t mean it’s usable.While testing, I found that the real metric that matters isn’t model size or even RAM usage, it’s actually…
Excerpt limited to ~120 words for fair-use compliance. The full article is at It's FOSS.
Discussion
0 commentsMore from It's FOSS
