electron-rare/turboquant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dbf85683e646b9501fdf262e8b751268ca0d0593
turboquant/REVIEW_PAPER_VS_IMPL.html
T
seroxdesign dbf85683e6 TurboQuant v0.2.0: modular architecture, MoE validation, full benchmarks 
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874).

Core:
- TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant)
- Modular architecture: capture, store, score, integration/vllm
- vLLM monkey-patch with free_kv_cache and hybrid decode
- 3 fused Triton kernels for decode attention

Validated on:
- RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity
- 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context
  - 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT
  - 30.9% KV savings (4.4x on full-attn layers, 1.45x overall)
  - 5/5 needle retrieval at max context

35 tests pass (19 modular + 7 core + 9 paper validation).
Adversarial audit included with honest assessment of all claims.
2026-03-27 13:44:07 -04:00

295 lines
14 KiB
HTML
Raw Blame History

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>TurboQuant: Paper vs Implementation</title>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
<style>
:root {
--bg: #0d1117; --surface: #161b22; --raised: #1c2129;
--border: #30363d; --text: #e6edf3; --muted: #8b949e;
--accent: #58a6ff; --green: #3fb950; --red: #f85149; --yellow: #d29922;
--green-bg: rgba(63,185,80,0.10); --red-bg: rgba(248,81,73,0.10); --yellow-bg: rgba(210,153,34,0.10);
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: var(--bg); color: var(--text); line-height: 1.55; }
.hdr { border-bottom: 1px solid var(--border); padding: 40px 24px 32px; text-align: center; }
.hdr h1 { font-size: 28px; font-weight: 700; letter-spacing: -0.4px; }
.hdr h1 span { color: var(--accent); }
.hdr .sub { color: var(--muted); font-size: 14px; margin-top: 6px; }
.wrap { max-width: 960px; margin: 0 auto; padding: 24px 20px 48px; }
.s { background: var(--surface); border: 1px solid var(--border); border-radius: 10px; padding: 24px; margin-bottom: 20px; }
.s h2 { font-size: 19px; font-weight: 700; margin-bottom: 16px; padding-bottom: 10px; border-bottom: 1px solid var(--border); }
h3 { font-size: 15px; font-weight: 600; margin: 16px 0 8px; }
p, li { font-size: 14px; } p { margin-bottom: 10px; }
ul, ol { padding-left: 18px; margin-bottom: 10px; } li { margin-bottom: 3px; }
strong { color: #fff; }
code { font-family: 'SF Mono', monospace; font-size: 12px; background: var(--raised); padding: 1px 5px; border-radius: 3px; border: 1px solid var(--border); }
table { width: 100%; border-collapse: collapse; margin: 12px 0; font-size: 13px; }
th { text-align: left; padding: 8px 10px; background: var(--raised); border: 1px solid var(--border); font-weight: 600; font-size: 12px; text-transform: uppercase; letter-spacing: 0.3px; color: var(--muted); }
td { padding: 8px 10px; border: 1px solid var(--border); }
.st { display: inline-block; font-size: 11px; font-weight: 600; padding: 2px 6px; border-radius: 3px; }
.ok { background: var(--green-bg); color: var(--green); }
.warn { background: var(--yellow-bg); color: var(--yellow); }
.fail { background: var(--red-bg); color: var(--red); }
.badge { display: inline-block; font-size: 10px; font-weight: 600; text-transform: uppercase; padding: 2px 6px; border-radius: 4px; margin-left: 6px; vertical-align: middle; }
.mw { background: #fff; border-radius: 6px; padding: 16px 12px; margin: 12px 0; overflow-x: auto; }
.g2 { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin: 12px 0; }
.g2 .c { background: var(--raised); border: 1px solid var(--border); border-radius: 6px; padding: 16px; }
.c h3 { margin-top: 0; }
.c ul { margin-bottom: 0; }
.cw { border-left: 3px solid; padding: 10px 14px; border-radius: 0 6px 6px 0; margin: 12px 0; font-size: 13px; }
.cw-r { border-color: var(--red); background: var(--red-bg); }
@media (max-width: 640px) { .g2 { grid-template-columns: 1fr; } .hdr h1 { font-size: 22px; } .s { padding: 16px; } }
</style>
</head>
<body>
<div class="hdr">
<h1><span>TurboQuant</span>: Paper vs Implementation</h1>
<p class="sub">arXiv:2504.19874 (ICLR 2026) &mdash; Qwen3.5-27B-AWQ on RTX 5090 &mdash; <a href="https://github.com/0xSero/turboquant" style="color:var(--accent)">github.com/0xSero/turboquant</a></p>
</div>
<div class="wrap">
<!-- A: Implementation Status -->
<div class="s">
<h2>Implementation Status vs Paper</h2>
<table>
<tr><th>Paper Component</th><th>Status</th><th>File</th></tr>
<tr><td>Random orthogonal rotation (QR)</td><td><span class="st ok">Faithful</span></td><td><code>rotation.py</code></td></tr>
<tr><td>Lloyd-Max codebook on Beta PDF</td><td><span class="st ok">Faithful</span></td><td><code>codebook.py</code></td></tr>
<tr><td>TurboQuant_MSE (Algorithm 1)</td><td><span class="st ok">Faithful</span></td><td><code>quantizer.py</code></td></tr>
<tr><td>TurboQuant_Prod (Algorithm 2)</td><td><span class="st ok">Faithful</span></td><td><code>quantizer.py</code></td></tr>
<tr><td>QJL projection + sign packing</td><td><span class="st ok">Faithful</span></td><td><code>quantizer.py</code>, <code>rotation.py</code></td></tr>
<tr><td>Value quantization</td><td><span class="st warn">Not from paper</span></td><td><code>kv_cache.py</code> (group quant)</td></tr>
<tr><td>Outlier channel splitting</td><td><span class="st fail">Removed</span></td><td>Was dead code, stripped</td></tr>
<tr><td>MLA support</td><td><span class="st fail">Stub</span></td><td><code>integration/vllm.py</code></td></tr>
<tr><td>vLLM integration + Triton kernels</td><td><span class="st ok">Novel eng.</span></td><td><code>integration/</code>, <code>triton_kernels.py</code></td></tr>
<tr><td>Modular capture/store/score</td><td><span class="st ok">Novel eng.</span></td><td><code>capture.py</code>, <code>store.py</code>, <code>score.py</code></td></tr>
</table>
</div>
<!-- B: Attention Backends -->
<div class="s">
<h2>Attention Backends</h2>
<table>
<tr><th>Backend</th><th>Status</th><th>Notes</th></tr>
<tr><td>Flash Attention (MHA/GQA)</td><td><span class="st ok">Working</span></td><td>Primary path, hooks on FlashAttentionImpl</td></tr>
<tr><td>Flash Attention (no-alloc prefill)</td><td><span class="st warn">Degraded</span></td><td>Falls back to F.scaled_dot_product_attention</td></tr>
<tr><td>FlashInfer</td><td><span class="st fail">Incompatible</span></td><td>Metadata mismatch with FP8 KV</td></tr>
<tr><td>MLA</td><td><span class="st fail">Stub</span></td><td>Passthrough only</td></tr>
<tr><td>GDN / Mamba</td><td><span class="st warn">N/A</span></td><td>No KV cache, TQ hooks on attention layers only</td></tr>
</table>
</div>
<!-- C: Known Issues -->
<div class="s">
<h2>Known Issues <span class="badge" style="background:var(--red-bg);color:var(--red);">After Trim</span></h2>
<ul>
<li><strong>No-alloc decode fallback returns zeros</strong> when TQ store has &lt;16 tokens</li>
<li><strong>No-alloc prefill uses <code>repeat_interleave</code></strong> for GQA, massive memory at 200k</li>
<li><strong>Value group_size hardcoded to 32</strong> (now explicit, was previously fragile math)</li>
<li><strong>MLA path is passthrough</strong> &mdash; no quantization for DeepSeek V3 etc.</li>
<li><strong>200k baseline stalls</strong> &mdash; no dual-case telemetry available</li>
<li><strong>Quality benchmarks are synthetic</strong> &mdash; no LongBench/RULER runs</li>
</ul>
</div>
<!-- D: Validation Results -->
<div class="s">
<h2>Validation Results (validate_paper.py) <span class="badge" style="background:var(--green-bg);color:var(--green);">9/9 pass</span></h2>
<table>
<tr><th>Test</th><th>What it validates</th><th>Result</th></tr>
<tr><td>MSE distortion bounds</td><td>Theorem 1: MSE &le; &radic;3&pi;/2 &middot; 1/4<sup>b</sup></td><td><span class="st ok">Pass</span> (b=1..4)</td></tr>
<tr><td>Codebook Table 1</td><td>Lloyd-Max MSE matches paper values</td><td><span class="st ok">Pass</span></td></tr>
<tr><td>Unbiasedness</td><td>Theorem 2: E[&langle;y, &tilde;x&rangle;] = &langle;y, x&rangle;</td><td><span class="st ok">Pass</span> (bits=2,3,4)</td></tr>
<tr><td>Distortion scaling</td><td>Theorem 3: 1/4<sup>b</sup> scaling</td><td><span class="st ok">Pass</span> (ratios &gt;2x per bit)</td></tr>
<tr><td>Recall@8 (N=4096)</td><td>Attention ranking quality at scale</td><td><span class="st ok">Pass</span></td></tr>
<tr><td>Rank correlation (N=2048)</td><td>Spearman corr 3-bit &gt;0.75, 4-bit &gt;0.90</td><td><span class="st ok">Pass</span></td></tr>
<tr><td>Needle@5 depths (N=4096)</td><td>Retrieval at 10/25/50/75/90% depth</td><td><span class="st ok">Pass</span> (3+4 bit)</td></tr>
<tr><td>Needle chunked (N=8192)</td><td>Multi-chunk retrieval</td><td><span class="st ok">Pass</span></td></tr>
<tr><td>Compression ratio</td><td>&gt;2x vs FP16</td><td><span class="st ok">Pass</span> (5.1x)</td></tr>
</table>
</div>
<!-- E: What was stripped -->
<div class="s">
<h2>What Was Stripped</h2>
<div class="g2">
<div class="c">
<h3 style="color:var(--red);">Removed</h3>
<ul>
<li><code>n_outlier_channels</code> / <code>outlier_key_bits</code> params (dead code)</li>
<li><code>_effective_bits()</code> method (never called)</li>
<li><code>_expand_query_for_gqa()</code> (dead function)</li>
<li><code>try_fused_decode()</code> (never called from working paths)</li>
<li>750+ lines of legacy classes in <code>vllm_attn_backend.py</code></li>
<li>Dead <code>console_scripts</code> entry point in <code>setup.py</code></li>
<li>Unused imports (<code>F</code>, <code>Tuple</code>, <code>MSEQuantized</code>, etc.)</li>
</ul>
</div>
<div class="c">
<h3 style="color:var(--green);">Fixed</h3>
<ul>
<li>Missing <code>import os</code> in <code>test_triton_kernels.py</code></li>
<li>Fragile group_size calc in <code>score.py</code> &rarr; explicit <code>32</code></li>
<li><code>vllm_attn_backend.py</code> reduced from 756 to ~190 lines (thin shim)</li>
</ul>
</div>
</div>
</div>
<!-- Diagram 1: System Architecture -->
<div class="s">
<h2>Architecture</h2>
<div class="mw">
<pre class="mermaid">
graph TB
subgraph "Core Quantization"
CB["codebook.py<br/>Lloyd-Max on Beta PDF"]
ROT["rotation.py<br/>QR rotation + QJL matrix"]
QMSE["TurboQuantMSE<br/>Rotate -> Quantize -> Pack"]
QPROD["TurboQuantProd<br/>MSE@(b-1) + QJL residual"]
end
subgraph "Serving Stack"
CAP["capture.py<br/>RingBuffer + KVCaptureEngine"]
STORE["store.py<br/>CompressedKVStore"]
SCORE["score.py<br/>compute_hybrid_attention"]
end
subgraph "vLLM Integration"
VLLM["integration/vllm.py<br/>off | capture_only | hybrid"]
SHIM["vllm_attn_backend.py<br/>enable_no_alloc shim"]
end
CB --> QMSE
ROT --> QMSE
ROT --> QPROD
QMSE --> QPROD
QPROD --> STORE
CAP --> STORE
STORE --> SCORE
VLLM --> CAP
VLLM --> SCORE
SHIM --> VLLM
</pre>
</div>
</div>
<!-- Diagram 2: Quantization Pipeline -->
<div class="s">
<h2>Quantization Pipeline</h2>
<div class="mw">
<pre class="mermaid">
graph LR
subgraph "Key: TurboQuant_Prod"
K["Key (H, T, D)"] --> NORM["||x||, normalize"]
NORM --> ROT["y = x @ Pi_T"]
ROT --> IDX["searchsorted -> indices"]
IDX --> PACK["bit-pack (b-1 bits)"]
ROT --> DQ["dequant MSE"]
DQ --> RES["r = x - x_mse"]
RES --> QJL["sign(r @ S_T)"]
QJL --> SIGNS["pack signs (1 bit/coord)"]
RES --> RNORM["||r||"]
end
subgraph "Value: Group Quant"
V["Value (H, T, D)"] --> GRP["reshape to groups"]
GRP --> MM["per-group min/max"]
MM --> VQ["round + bit-pack (2-bit)"]
end
</pre>
</div>
</div>
<!-- Diagram 3: Read/Write Paths -->
<div class="s">
<h2>Read / Write Paths</h2>
<div class="mw">
<pre class="mermaid">
graph TB
subgraph "Write"
PRE["Prefill N tokens"] --> SP{"N > ring?"}
SP -->|Yes| CMP["Compress first N-ring -> store"]
SP -->|Yes| BUF["Last ring tokens -> ring buffer"]
SP -->|No| BA["All -> ring buffer"]
DEC["Decode 1 token"] --> RW["ring.write"]
RW --> OV{"Overflow?"}
OV -->|Yes| FL["Drain -> store.append_chunk"]
end
subgraph "Read (Decode)"
Q["Query"] --> HH{"History >= 16?"}
HH -->|No| EX["Attend exact buffer only"]
HH -->|Yes| HY["Hybrid: dequant history + cat recent"]
HY --> ATT["einsum attention with GQA broadcast"]
end
</pre>
</div>
</div>
<!-- Diagram 4: vLLM Hook Chain -->
<div class="s">
<h2>vLLM Hook Chain</h2>
<div class="mw">
<pre class="mermaid">
sequenceDiagram
participant U as User
participant E as enable_no_alloc
participant X as Executor
participant I as FlashAttentionImpl
participant T as TQ LayerState
U->>E: enable_no_alloc(bits=3)
E->>X: patch get_kv_cache_specs
U->>X: LLM() init
X->>I: install_hooks (16 layers)
I->>T: create store + engine per layer
Note over I: Share KV cache across TQ layers
U->>X: generate(prompt)
Note over I: PREFILL
I->>T: SDPA attention + capture K/V
Note over I: DECODE
I->>T: compute_hybrid_attention
T-->>I: output
</pre>
</div>
</div>
<!-- Summary -->
<div class="s">
<h2>Summary</h2>
<div class="g2">
<div class="c">
<h3 style="color:var(--green);">What Works</h3>
<ul>
<li>Core TQ_MSE + TQ_Prod faithful to paper</li>
<li>All 3 theorems validated empirically</li>
<li>Needle retrieval passes at 8192 tokens</li>
<li>5.1x compression vs FP16</li>
<li>30k A/B: +5.7% prefill, +3.1% output speed</li>
<li>200k TQ completed (199,952 tokens)</li>
</ul>
</div>
<div class="c">
<h3 style="color:var(--red);">What Doesn't</h3>
<ul>
<li>MLA: stub only (DeepSeek V3 unsupported)</li>
<li>FlashInfer: incompatible</li>
<li>200k quality unvalidated (no LongBench)</li>
<li>Multi-sequence batching untested</li>
<li>No-alloc prefill slow (SDPA, not flash)</li>
</ul>
</div>
</div>
<div class="cw cw-r">
<strong>Key gap:</strong> The "identical output" claim holds only for trivial prompts. Long-context retrieval quality is unquantified against standard benchmarks (LongBench, RULER, Needle-in-Haystack at 200k). The 200k needle failure in the handoff doc is concerning.
</div>
</div>
</div>
<script>
mermaid.initialize({ startOnLoad: true, theme: 'default', securityLevel: 'loose',
flowchart: { useMaxWidth: true, curve: 'basis' },
sequence: { useMaxWidth: true, wrap: true }
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink