How to Run LLMs Locally (Great For Learning and Privacy)

The local LLM tooling landscape has matured into a clear tiered stack. At the bottom, llama.cpp provides a lightweight C++ inference engine that runs on CPU, GPU, or Apple Silicon, using the GGUF format with aggressive quantization to fit models onto consumer hardware. Most teams will never touch it directly, but they need to know it exists because it underpins nearly everything else and is the fallback for constrained deployments.

For developer prototyping, Ollama has become the default starting point. It wraps llama.cpp, handles model selection and download automatically, and exposes an OpenAI-compatible API. This matters operationally because it means a developer can swap from a paid hosted endpoint to a local model by changing a single base URL, enabling offline iteration and privacy-sensitive development without workflow changes. The tradeoff is that Ollama is not built for concurrent multi-user serving; it's a single-user prototyping tool.

LM Studio fills the GUI gap, targeting the non-terminal user who wants to browse, download, and compare models visually. Its practical value is in model evaluation: showing hardware requirements, quantization options, and memory warnings before download. This reduces the friction of testing whether an open model fits a specific task on specific hardware, which is useful for tech leads evaluating what can move off paid APIs.

On the production serving side, vLLM and SGLang represent the state of the art. vLLM uses paged attention to reduce KV cache memory waste and continuous batching to eliminate GPU idle time between requests. These are not incremental improvements; they fundamentally change the throughput economics of self-hosting LLMs. SGLang's Radix Attention adds a tree-based prefix cache, making it particularly efficient for RAG and multi-turn chat workloads where prompts share long common prefixes. The video notes SGLang is used in production by xAI and DeepSeek deployments, which lends credibility. The engineering decision between vLLM and SGLang will depend on workload shape: prefix-heavy workloads favor SGLang; general high-throughput serving defaults to vLLM.

For Apple Silicon shops, MLX LM exploits the unified memory architecture of M-series chips. A Mac Studio with 192GB unified memory can load models that would require multiple expensive discrete GPUs on x86. This is a niche but real capability for teams already on Apple hardware. The watchpoint: MLX LM is Apple-specific and has a smaller community than the CUDA-centric alternatives.

The unstated message is that the local LLM stack now mirrors the maturity curve of databases or message queues: you pick the tool for the stage. What's new is not any single tool, but that the entire pipeline—from evaluation to production serving—is now possible without a hosted API dependency.