 seroxdesign
|
7ac9b8d165
|
Strip to bare minimum, switch license to GPL-3.0
Remove 30 scratch/experimental files: HTML reports, profiling scripts,
validation scripts, autoresearch harness, tests, and benchmarks.
Keep only: turboquant/ package, proof.py, benchmark.py, setup.py, README.
License changed from MIT to GPL-3.0.
|
2026-03-27 18:28:25 -04:00 |
|
|
seroxdesign
|
dbf85683e6
|
TurboQuant v0.2.0: modular architecture, MoE validation, full benchmarks
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874).
Core:
- TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant)
- Modular architecture: capture, store, score, integration/vllm
- vLLM monkey-patch with free_kv_cache and hybrid decode
- 3 fused Triton kernels for decode attention
Validated on:
- RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity
- 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context
- 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT
- 30.9% KV savings (4.4x on full-attn layers, 1.45x overall)
- 5/5 needle retrieval at max context
35 tests pass (19 modular + 7 core + 9 paper validation).
Adversarial audit included with honest assessment of all claims.
|
2026-03-27 13:44:07 -04:00 |
|