Local AI Works, but Your Coding Agent Harness Matters More

Open-source models like GLM 5.2 now match frontier performance. Tools—llama.cpp's Web UI, LM Studio, Hugging Face inference—lower local deployment barriers. Quantization (dynamic, QAT, MTP) shrinks a 1.5TB model to 217GB with little accuracy loss, making it feasible on a MacBook. MoE architectures (Gemma, Qwen) are especially efficient.

Coding agents (Claude Code, Pi, Open Code) now support open models, but early demos reveal that different harnesses produce qualitatively different outputs from the same model. Workflow design, not just model selection, determines productivity. This suggests teams must test multiple harnesses before locking into a pipeline.

Cost is no longer purely API fees: upfront hardware investment and VRAM planning replace subscriptions, but smaller local models may falter on long-running, complex tasks. MTP can double inference speed with zero accuracy loss, improving the local experience, yet not all models support it.

Security presents a double-edged sword: local models keep data private, but their smaller size may increase vulnerability to prompt injection. No comparable guardrails exist yet. Engineering leaders must weigh privacy against risk and plan for additional governance layers as local AI adoption scales.