electron-rare/turboquant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dbf85683e646b9501fdf262e8b751268ca0d0593
turboquant/explainer.html
T
seroxdesign dbf85683e6 TurboQuant v0.2.0: modular architecture, MoE validation, full benchmarks 
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874).

Core:
- TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant)
- Modular architecture: capture, store, score, integration/vllm
- vLLM monkey-patch with free_kv_cache and hybrid decode
- 3 fused Triton kernels for decode attention

Validated on:
- RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity
- 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context
  - 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT
  - 30.9% KV savings (4.4x on full-attn layers, 1.45x overall)
  - 5/5 needle retrieval at max context

35 tests pass (19 modular + 7 core + 9 paper validation).
Adversarial audit included with honest assessment of all claims.
2026-03-27 13:44:07 -04:00

299 lines
17 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>TurboQuant — How It Works</title>
<style>
:root { --bg: #0f0f1a; --card: #1a1a2e; --accent: #4fc3f7; --green: #66bb6a; --red: #ef5350; --yellow: #ffd54f; --text: #e0e0e0; --dim: #888; }
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: 'Segoe UI', system-ui, -apple-system, sans-serif; background: var(--bg); color: var(--text); line-height: 1.6; }
.container { max-width: 960px; margin: 0 auto; padding: 40px 24px; }
h1 { font-size: 2.8rem; text-align: center; margin-bottom: 8px; }
h1 span { color: var(--accent); }
.subtitle { text-align: center; color: var(--dim); font-size: 1.1rem; margin-bottom: 48px; }
h2 { font-size: 1.6rem; margin: 48px 0 20px; padding-bottom: 8px; border-bottom: 2px solid #333; }
h3 { font-size: 1.15rem; color: var(--accent); margin: 16px 0 8px; }
p, li { font-size: 0.95rem; }
ul { padding-left: 20px; margin: 8px 0; }
li { margin: 4px 0; }
.hero-stats { display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; margin: 32px 0; }
.stat { background: var(--card); border-radius: 12px; padding: 24px; text-align: center; border: 1px solid #333; }
.stat .num { font-size: 2.4rem; font-weight: 800; }
.stat .label { font-size: 0.85rem; color: var(--dim); margin-top: 4px; }
.stat.green .num { color: var(--green); }
.stat.red .num { color: var(--red); }
.stat.blue .num { color: var(--accent); }
.flow { display: flex; align-items: stretch; gap: 0; margin: 24px 0; flex-wrap: wrap; justify-content: center; }
.flow-step { background: var(--card); border: 1px solid #333; border-radius: 10px; padding: 16px; flex: 1; min-width: 140px; max-width: 200px; text-align: center; position: relative; }
.flow-step .icon { font-size: 1.8rem; margin-bottom: 6px; }
.flow-step .title { font-weight: 700; font-size: 0.9rem; }
.flow-step .desc { font-size: 0.78rem; color: var(--dim); margin-top: 4px; }
.flow-arrow { display: flex; align-items: center; font-size: 1.4rem; color: var(--dim); padding: 0 6px; }
.two-col { display: grid; grid-template-columns: 1fr 1fr; gap: 24px; margin: 16px 0; }
@media (max-width: 700px) { .two-col { grid-template-columns: 1fr; } .hero-stats { grid-template-columns: 1fr; } }
.card { background: var(--card); border: 1px solid #333; border-radius: 10px; padding: 20px; }
.card h3 { margin-top: 0; }
.diagram { background: #111; border: 1px solid #333; border-radius: 10px; padding: 24px; margin: 20px 0; font-family: 'Courier New', monospace; font-size: 0.82rem; line-height: 1.8; overflow-x: auto; white-space: pre; color: #ccc; }
.diagram .hl { color: var(--accent); font-weight: 700; }
.diagram .gr { color: var(--green); }
.diagram .rd { color: var(--red); }
.diagram .yw { color: var(--yellow); }
table { width: 100%; border-collapse: collapse; margin: 16px 0; font-size: 0.9rem; }
th { background: #222; padding: 10px 12px; text-align: left; font-weight: 600; border-bottom: 2px solid #444; }
td { padding: 8px 12px; border-bottom: 1px solid #2a2a2a; }
tr:hover td { background: #1a1a2e; }
.g { color: var(--green); font-weight: 700; }
.r { color: var(--red); }
.math { font-family: 'Times New Roman', serif; font-style: italic; font-size: 1.05em; }
.code { background: #222; padding: 2px 6px; border-radius: 4px; font-family: monospace; font-size: 0.85em; }
.section-num { display: inline-block; background: var(--accent); color: #000; font-weight: 800; width: 28px; height: 28px; border-radius: 50%; text-align: center; line-height: 28px; font-size: 0.85rem; margin-right: 8px; vertical-align: middle; }
.tag { display: inline-block; padding: 2px 10px; border-radius: 12px; font-size: 0.75rem; font-weight: 600; }
.tag-green { background: #1b5e20; color: #a5d6a7; }
.tag-red { background: #b71c1c; color: #ef9a9a; }
.tag-blue { background: #0d47a1; color: #90caf9; }
</style>
</head>
<body>
<div class="container">
<h1>Turbo<span>Quant</span></h1>
<div class="subtitle">KV Cache Compression for vLLM &mdash; 2&times; context, 0% overhead, 30 GB freed</div>
<div class="hero-stats">
<div class="stat green"><div class="num">30 GB</div><div class="label">KV cache freed (4 GPUs)</div></div>
<div class="stat blue"><div class="num">2.0&times;</div><div class="label">context capacity</div></div>
<div class="stat red"><div class="num">0%</div><div class="label">throughput overhead</div></div>
</div>
<!-- ─── THE PROBLEM ─── -->
<h2><span class="section-num">1</span> The Problem</h2>
<div class="two-col">
<div class="card">
<h3>KV cache grows with every token</h3>
<p>During generation, every layer stores a <strong>Key</strong> and <strong>Value</strong> vector for every previous token. For long contexts this dominates GPU memory.</p>
<div class="diagram">Token 1 &rarr; K₁ V₁
Token 2 &rarr; K₁ V₁ K₂ V₂
Token 3 &rarr; K₁ V₁ K₂ V₂ K₃ V₃
...
<span class="rd">Token N &rarr; K₁..Kₙ V₁..Vₙ &larr; O(N) memory</span></div>
</div>
<div class="card">
<h3>Qwen3.5-27B on 4&times; RTX 3090</h3>
<ul>
<li>Model weights: ~13 GB / GPU</li>
<li><strong>KV cache: ~7.5 GB / GPU</strong> (30 GB total)</li>
<li>Max context: 457,072 tokens</li>
<li>Want longer? Need more GPUs &mdash; expensive</li>
</ul>
<p style="margin-top:12px; color: var(--yellow);">What if we could compress the KV cache to 39% of its size?</p>
</div>
</div>
<!-- ─── HIGH-LEVEL PIPELINE ─── -->
<h2><span class="section-num">2</span> The TurboQuant Pipeline</h2>
<p>Each KV pair goes through a compression pipeline that reduces <strong>512 bytes &rarr; 198 bytes</strong> per token (2.6&times; compression):</p>
<div class="flow">
<div class="flow-step"><div class="icon">🔑</div><div class="title">Raw Key</div><div class="desc">256-dim bf16<br>512 bytes</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--accent);"><div class="icon">🔄</div><div class="title">Rotate</div><div class="desc">Random orthogonal<br>matrix &Pi;</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--accent);"><div class="icon">📊</div><div class="title">Quantize</div><div class="desc">3-bit Lloyd-Max<br>8 centroids</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--accent);"><div class="icon">±</div><div class="title">QJL Signs</div><div class="desc">Residual direction<br>sign bits</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--green);"><div class="icon">✅</div><div class="title">Compressed</div><div class="desc">~110 bytes<br><strong>4.7&times; smaller</strong></div></div>
</div>
<div class="flow" style="margin-top: 16px;">
<div class="flow-step"><div class="icon">💎</div><div class="title">Raw Value</div><div class="desc">256-dim bf16<br>512 bytes</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--yellow);"><div class="icon">📦</div><div class="title">Group Quantize</div><div class="desc">2-bit per value<br>groups of 32</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--yellow);"><div class="icon">🗜️</div><div class="title">Bit-Pack</div><div class="desc">4 values per byte<br>+ scales & zeros</div></div>
<div class="flow-arrow">&rarr;</div>
<div class="flow-step" style="border-color: var(--green);"><div class="icon">✅</div><div class="title">Compressed</div><div class="desc">~88 bytes<br><strong>5.8&times; smaller</strong></div></div>
</div>
<!-- ─── KEY COMPRESSION DETAIL ─── -->
<h2><span class="section-num">3</span> Key Compression (3-bit)</h2>
<div class="two-col">
<div class="card">
<h3>Step A: Random Rotation</h3>
<p>Multiply keys by a random orthogonal matrix <span class="math">&Pi;</span>. This spreads information uniformly across dimensions so no single dimension carries disproportionate signal.</p>
<p style="margin-top:8px;"><span class="math">K<sub>rot</sub> = K &middot; &Pi;</span> &nbsp;&nbsp; (norm-preserving)</p>
<h3 style="margin-top:16px;">Step B: Lloyd-Max Quantization</h3>
<p>After rotation, each dimension follows a <strong>Beta distribution</strong>. Lloyd-Max finds the <strong>MSE-optimal</strong> 8 centroids for this distribution.</p>
<p style="margin-top:8px;">Codebooks are <strong>pre-computed offline</strong> for d=256 &mdash; zero runtime overhead.</p>
</div>
<div class="card">
<h3>Step C: QJL Residual Signs</h3>
<p>The quantization residual <span class="math">r = K<sub>rot</sub> - K&#x0302;</span> still contains directional information. QJL (Quantized Johnson-Lindenstrauss) projects it into a compact sign vector:</p>
<p style="margin-top:8px;"><span class="math">signs = sign(S &middot; r)</span></p>
<p>where <span class="math">S</span> is a random &plusmn;1 matrix. This preserves inner-product relationships with high probability.</p>
<h3 style="margin-top:16px;">Storage Breakdown</h3>
<table>
<tr><td>3-bit indices (256 dims)</td><td style="text-align:right;">96 B</td></tr>
<tr><td>Norm (fp32)</td><td style="text-align:right;">4 B</td></tr>
<tr><td>Residual norm (fp32)</td><td style="text-align:right;">4 B</td></tr>
<tr><td>QJL signs</td><td style="text-align:right;">~8 B</td></tr>
<tr><th>Total per token per head</th><th style="text-align:right;">~110 B</th></tr>
</table>
</div>
</div>
<!-- ─── VALUE COMPRESSION DETAIL ─── -->
<h2><span class="section-num">4</span> Value Compression (2-bit)</h2>
<div class="card">
<div class="two-col">
<div>
<h3>Group Quantization</h3>
<p>Values are split into groups of 32 dimensions. Each group gets its own scale and zero point, then each value is mapped to one of <strong>4 levels</strong> (2 bits).</p>
<div class="diagram"><span class="yw">For group [d₀ .. d₃₁]:</span>
scale = (max - min) / 3
zero = min
idx_i = round((val_i - zero) / scale) <span class="hl">&larr; 0,1,2,3</span>
<span class="yw">Bit-packing (4 values per byte):</span>
byte = idx₀ | (idx₁&lt;&lt;2) | (idx₂&lt;&lt;4) | (idx₃&lt;&lt;6)
<span class="gr">256 dims &rarr; 64 packed bytes + 16B scales + 16B zeros = 88 bytes</span></div>
</div>
<div>
<h3>Why 2-bit works for values</h3>
<ul>
<li>Values are <strong>summed with softmax weights</strong> during attention &mdash; small errors average out across many tokens</li>
<li>Per-group scales preserve the <strong>dynamic range</strong> of each dimension cluster</li>
<li>4&times; denser than 8-bit quantization</li>
</ul>
<h3 style="margin-top:16px;">Combined Compression</h3>
<table>
<tr><th>Component</th><th>Original</th><th>Compressed</th></tr>
<tr><td>Key</td><td class="r">512 B</td><td class="g">110 B</td></tr>
<tr><td>Value</td><td class="r">512 B</td><td class="g">88 B</td></tr>
<tr><th>Total</th><th class="r">1,024 B</th><th class="g">198 B (2.6&times;)</th></tr>
</table>
</div>
</div>
</div>
<!-- ─── VLLM INTEGRATION ─── -->
<h2><span class="section-num">5</span> vLLM Integration</h2>
<div class="diagram" style="font-size: 0.78rem; line-height: 2;">
<span class="yw">═══ PREFILL (processing the prompt) ═════════════════════════════════════════</span>
User prompt &rarr; vLLM tokenizer &rarr; <span class="hl">FlashAttention forward(Q, K, V)</span>
&darr;
Writes K,V to paged KV cache (normal)
&darr;
<span class="gr">TQ Hook intercepts K,V &rarr; Rotate &rarr; Quantize &rarr; Store compressed</span>
&darr;
Returns attention output (from flash)
<span class="yw">═══ DECODE (generating each new token) ══════════════════════════════════════</span>
New token &rarr; vLLM &rarr; <span class="gr">TQ ACTIVE mode intercepts forward()</span>
&darr;
<span class="gr">Reconstruct K,V from compressed store (Triton kernel)</span>
<span class="gr">Compute attention entirely in TQ</span>
<span class="gr">Flash attention SKIPPED &mdash; paged cache not read</span>
&darr;
Returns attention output
<span class="yw">═══ FREE KV CACHE ═══════════════════════════════════════════════════════════</span>
After prefill, TQ compressed store has all token data.
<span class="rd">Replace paged KV cache tensors with 1-byte dummies.</span>
<span class="gr">torch.cuda.empty_cache() &rarr; 30 GB VRAM returned to pool!</span>
Decode continues using only TQ compressed store.
</div>
<!-- ─── DECODE KERNEL ─── -->
<h2><span class="section-num">6</span> Fused Triton Decode Kernel</h2>
<div class="two-col">
<div class="card">
<h3>What happens every decode step</h3>
<p>For each new query token <span class="math">q</span>, against all <span class="math">N</span> cached tokens:</p>
<ol style="padding-left: 20px; margin-top: 8px;">
<li><strong>Dequantize keys:</strong> indices &rarr; centroids, undo rotation via <span class="math">&Pi;<sup>T</sup></span></li>
<li><strong>Attention scores:</strong> <span class="math">&alpha;<sub>i</sub> = q &middot; k<sub>i</sub> / &radic;d</span></li>
<li><strong>Dequantize values:</strong> unpack 2-bit &rarr; scale &middot; idx + zero</li>
<li><strong>Weighted sum:</strong> <span class="math">out = &sum; softmax(&alpha;<sub>i</sub>) &middot; v<sub>i</sub></span></li>
</ol>
<p style="margin-top: 12px;"><strong>All 4 steps fused into ONE Triton kernel</strong> &mdash; no intermediate tensors allocated.</p>
</div>
<div class="card">
<h3>Why zero overhead</h3>
<ul>
<li>Decode is <strong>compute-bound</strong>, not memory-bound</li>
<li>Reading compressed data is <strong>faster</strong> (less memory bandwidth)</li>
<li>Dequantization is cheap ALU ops fused into the attention loop</li>
<li>Works with <strong>GQA</strong> (Grouped-Query Attention)</li>
</ul>
<table style="margin-top: 16px;">
<tr><th>Metric</th><th>Baseline</th><th>TQ</th></tr>
<tr><td>Throughput</td><td>6.0 tok/s</td><td class="g">6.0 tok/s</td></tr>
<tr><td>Overhead</td><td>&mdash;</td><td class="g">0%</td></tr>
<tr><td>Output</td><td>reference</td><td class="g">identical</td></tr>
</table>
</div>
</div>
<!-- ─── RESULTS ─── -->
<h2><span class="section-num">7</span> Results</h2>
<p style="text-align:center; color: var(--dim); margin-bottom: 16px;">Qwen3.5-27B &middot; 4&times; RTX 3090 &middot; TP=4 &middot; bf16 &middot; gpu_mem=0.90</p>
<table>
<tr><th>Metric</th><th>Baseline vLLM</th><th>TurboQuant</th><th>Change</th></tr>
<tr><td>KV cache blocks</td><td>583</td><td>583 &rarr; freed</td><td class="g">&minus;30 GB</td></tr>
<tr><td>VRAM / GPU</td><td>21,539 MB</td><td>21,299 MB</td><td class="g">&minus;240 MB visible</td></tr>
<tr><td>Tensor bytes freed / GPU</td><td>&mdash;</td><td>7,489 MB</td><td class="g">available in CUDA pool</td></tr>
<tr><td>Total freed</td><td>&mdash;</td><td><strong>30.0 GB</strong></td><td></td></tr>
<tr><td>Max token capacity</td><td>457,072</td><td><strong>914,144</strong></td><td class="g"><strong>2.0&times;</strong></td></tr>
<tr><td>Throughput</td><td>6.0 tok/s</td><td>6.0 tok/s</td><td class="g">0% overhead</td></tr>
<tr><td>Output quality</td><td>reference</td><td>identical</td><td>&mdash;</td></tr>
</table>
<!-- ─── ARCHITECTURE ─── -->
<h2><span class="section-num">8</span> Code Architecture</h2>
<div class="diagram" style="line-height: 2.2; font-size: 0.8rem;">turboquant/
<span class="hl">codebook.py</span> &larr; Lloyd-Max optimal quantizer for Beta distribution
<span class="hl">codebooks/</span> &larr; Pre-generated codebook files (d=256, bits 2/3/4)
<span class="hl">rotation.py</span> &larr; Random orthogonal &Pi; + QJL projection S
<span class="hl">quantizer.py</span> &larr; TurboQuantMSE + TurboQuantProd pipeline
<span class="hl">kv_cache.py</span> &larr; KV cache manager, value bit-packing
<span class="gr">triton_kernels.py</span> &larr; 3 fused Triton kernels (decode attention)
<span class="yw">vllm_attn_backend.py</span> &larr; Monkey-patch hooks, free_kv_cache(), enable_no_alloc()
<span class="rd">proof.py</span> &larr; Definitive A/B benchmark (separate processes)
<span class="rd">benchmark.py</span> &larr; Throughput + quality + VRAM benchmark</div>
<div style="text-align: center; margin-top: 48px; padding: 32px; border-top: 1px solid #333;">
<div style="font-size: 1.4rem; font-weight: 700;">Same model. Same GPUs. <span style="color: var(--accent);">2&times; the context.</span></div>
<div style="margin-top: 12px;"><a href="https://github.com/0xSero/turboquant" style="color: var(--accent); text-decoration: none;">github.com/0xSero/turboquant</a></div>
</div>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink