TurboQuant v0.2.0: modular architecture, MoE validation, full benchmarks
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874).
Core:
- TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant)
- Modular architecture: capture, store, score, integration/vllm
- vLLM monkey-patch with free_kv_cache and hybrid decode
- 3 fused Triton kernels for decode attention
Validated on:
- RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity
- 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context
- 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT
- 30.9% KV savings (4.4x on full-attn layers, 1.45x overall)
- 5/5 needle retrieval at max context
35 tests pass (19 modular + 7 core + 9 paper validation).
Adversarial audit included with honest assessment of all claims.
seroxdesign
dbf85683e6