Colibri
A 744B-parameter model on a 25GB laptop, in pure C
What it is
A pure-C inference engine that runs GLM-5.2 — a 744-billion-parameter mixture-of-experts model — on roughly 25GB of RAM by treating VRAM, RAM, and disk as one memory hierarchy and streaming expert weights from storage on demand. Implements MLA attention, speculative decoding, grammar-forced output, and optional CUDA/Metal.
Why it's interesting
A radical answer to 'frontier models need datacenters': correctness-first systems hacking in the llama.cpp spirit that makes a datacenter-class MoE technically runnable on consumer hardware — even at 0.05 tokens/second when cold.
Use cases
- Research and education on MoE offloading
- Batch or overnight generation where latency is irrelevant
- Proving a point about what consumer hardware can do
Who it's for
Systems programmers, local-inference enthusiasts, patient people
Setup
Advanced. ~25GB RAM minimum, a large fast NVMe drive holding hundreds of GB of expert weights, and a C toolchain
Limitations & cautions
Interactive use is impractical at cold-decode speeds, it currently targets one model family, and it is experimental through and through.
Editorial takeaway
Nobody needs this. That's not the point. The point is that the wall between consumer and datacenter hardware is made of engineering, not physics.