seroxdesign
dbf85683e6
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874). Core: - TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant) - Modular architecture: capture, store, score, integration/vllm - vLLM monkey-patch with free_kv_cache and hybrid decode - 3 fused Triton kernels for decode attention Validated on: - RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity - 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context - 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT - 30.9% KV savings (4.4x on full-attn layers, 1.45x overall) - 5/5 needle retrieval at max context 35 tests pass (19 modular + 7 core + 9 paper validation). Adversarial audit included with honest assessment of all claims.
70 lines
3.4 KiB
HTML
70 lines
3.4 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
<title>TurboQuant 5090 Results</title>
|
|
<style>
|
|
body { font-family: Inter, Arial, sans-serif; margin: 24px; background:#0b1020; color:#e8ecf3; }
|
|
h1,h2 { margin: 0 0 12px; }
|
|
h1 { font-size: 28px; }
|
|
h2 { margin-top: 28px; font-size: 20px; color:#9cc3ff; }
|
|
.meta { color:#b7c2d8; margin-bottom: 18px; }
|
|
table { border-collapse: collapse; width: 100%; margin: 12px 0 24px; background:#111831; }
|
|
th, td { border: 1px solid #2b355c; padding: 10px 12px; text-align: left; font-size: 14px; }
|
|
th { background:#182244; color:#cfe1ff; }
|
|
tr:nth-child(even) td { background:#0f1630; }
|
|
.good { color:#7ee787; font-weight: 600; }
|
|
.note { color:#b7c2d8; font-size: 14px; }
|
|
code { background:#17203d; padding:2px 6px; border-radius:6px; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<h1>TurboQuant on RTX 5090</h1>
|
|
<div class="meta">Model: <code>QuantTrio/Qwen3.5-27B-AWQ</code> · Runtime: <code>vllm==0.18.0</code> · GPU: <code>RTX 5090 32GB</code></div>
|
|
|
|
<h2>Summary</h2>
|
|
<table>
|
|
<tr><th>Metric</th><th>Baseline</th><th>TurboQuant</th></tr>
|
|
<tr><td>Status</td><td>init_ok</td><td>init_ok</td></tr>
|
|
<tr><td>KV cache blocks</td><td>57</td><td>57</td></tr>
|
|
<tr><td>Measured baseline context</td><td>44,688</td><td>-</td></tr>
|
|
<tr><td>Verified max working context</td><td>44,688</td><td class="good">81,920</td></tr>
|
|
<tr><td>Improvement</td><td>-</td><td class="good">1.83x</td></tr>
|
|
<tr><td>Hooks installed</td><td>-</td><td>16</td></tr>
|
|
<tr><td>Freed bytes</td><td>-</td><td>2,928,672,768</td></tr>
|
|
<tr><td>Freed GiB</td><td>-</td><td>2.73</td></tr>
|
|
<tr><td>Elapsed time (s)</td><td>126.07</td><td>57.65</td></tr>
|
|
</table>
|
|
|
|
<h2>GPU Memory</h2>
|
|
<table>
|
|
<tr><th>Phase</th><th>Used MiB</th><th>Free MiB</th></tr>
|
|
<tr><td>Baseline after init</td><td>27895</td><td>4218</td></tr>
|
|
<tr><td>Baseline after generation</td><td>28155</td><td>3958</td></tr>
|
|
<tr><td>TurboQuant after init</td><td>27895</td><td>4218</td></tr>
|
|
<tr><td>TurboQuant after generation</td><td>28173</td><td>3940</td></tr>
|
|
<tr><td>TurboQuant after free_kv_cache()</td><td>27915</td><td>4198</td></tr>
|
|
</table>
|
|
|
|
<h2>Capacity Search Results</h2>
|
|
<table>
|
|
<tr><th>max_model_len</th><th>Status</th><th>Hooks</th><th>Elapsed (s)</th><th>Sample Output</th></tr>
|
|
<tr><td>44,688</td><td class="good">ok</td><td>16</td><td>57.65</td><td>KV cache compression</td></tr>
|
|
<tr><td>49,152</td><td class="good">ok</td><td>16</td><td>69.66</td><td>KV cache compression</td></tr>
|
|
<tr><td>53,248</td><td class="good">ok</td><td>16</td><td>53.76</td><td>KV cache compression</td></tr>
|
|
<tr><td>57,344</td><td class="good">ok</td><td>16</td><td>63.03</td><td>KV cache compression</td></tr>
|
|
<tr><td>61,440</td><td class="good">ok</td><td>16</td><td>54.65</td><td>KV cache compression</td></tr>
|
|
<tr><td>65,536</td><td class="good">ok</td><td>16</td><td>54.02</td><td>KV cache compression</td></tr>
|
|
<tr><td>73,728</td><td class="good">ok</td><td>16</td><td>72.70</td><td>KV cache compression</td></tr>
|
|
<tr><td>81,920</td><td class="good">ok</td><td>16</td><td>62.59</td><td>KV cache compression</td></tr>
|
|
</table>
|
|
|
|
<h2>Notes</h2>
|
|
<div class="note">
|
|
<p>These values are from completed remote experiments only.</p>
|
|
<p>The TurboQuant repository code itself was not modified.</p>
|
|
</div>
|
|
</body>
|
|
</html>
|