A GitHub developer has published a comprehensive guide to running state-of-the-art language models locally, detailing hardware configurations from $2,000 to $46,000 that can match the performance of commercial AI services.

James O'Brien's setup centers on four NVIDIA RTX Pro 6000 GPUs with 96GB VRAM each, providing 384GB total memory to run models approaching Claude Opus-level performance. The configuration achieves 80 tokens per second on GLM-5.2-594B, a 594-billion parameter model.

The build uses an unconventional approach with PCIe4 switches from c-payne.com, allowing GPUs to communicate directly at wire speeds rather than routing through the PCI root complex. This reduces latency while avoiding expensive PCIe5 hardware costs.

"By using an older host-gen4 setup rather than gen5, we're saving about $10,000 in host costs while still getting very good performance," O'Brien wrote in the guide.

The base system runs on a last-generation AMD EPYC Milan processor with DDR4 memory, sourced primarily from eBay to control costs. The motherboard, CPU, and 128GB RAM total $1,861, while the four GPUs represent the bulk of the $46,000 investment.

Budget alternatives offer surprising capability

For developers with smaller budgets, O'Brien recommends a $2,000 configuration using two RTX 3090s with 48GB total VRAM. This setup can run Qwen3.6-27B and includes local speech-to-text capabilities using cohere-transcribe.

The guide includes ready-to-run Docker configurations and benchmarking tools. O'Brien's peer-to-peer GPU setup achieves 27.5GB/s bandwidth with sub-microsecond latency between cards.

The hardware approach addresses concerns about relying on commercial AI services while providing performance competitive with hosted solutions. O'Brien notes that "spilling over into system RAM makes LLM performance unusably slower for agentic workloads."

The complete guide includes BIOS configuration details, kernel parameters, and power management settings needed to run the high-end hardware on standard 110V circuits.