seroxdesign
dbf85683e6
KV cache compression for LLM inference (ICLR 2026, arXiv:2504.19874). Core: - TurboQuantProd: 3-bit keys (MSE + QJL), 2-bit/4-bit values (group quant) - Modular architecture: capture, store, score, integration/vllm - vLLM monkey-patch with free_kv_cache and hybrid decode - 3 fused Triton kernels for decode attention Validated on: - RTX 5090: Qwen3.5-27B-AWQ, 30GB KV freed, 2x context capacity - 8x RTX 3090: Qwen3.5-35B-A3B MoE at 131k context - 8,238 tok/s prefill, 98 tok/s decode, 15.9s TTFT - 30.9% KV savings (4.4x on full-attn layers, 1.45x overall) - 5/5 needle retrieval at max context 35 tests pass (19 modular + 7 core + 9 paper validation). Adversarial audit included with honest assessment of all claims.
32 lines
1001 B
Python
32 lines
1001 B
Python
from setuptools import setup, find_packages
|
|
|
|
setup(
|
|
name="turboquant",
|
|
version="0.1.0",
|
|
description="TurboQuant: Near-optimal KV cache quantization for LLM inference",
|
|
long_description=open("README.md").read(),
|
|
long_description_content_type="text/markdown",
|
|
author="Implementation based on Zandieh et al. (ICLR 2026)",
|
|
url="https://github.com/0xSero/turboquant",
|
|
packages=find_packages(),
|
|
package_data={"turboquant": ["codebooks/*.json"]},
|
|
python_requires=">=3.10",
|
|
install_requires=[
|
|
"torch>=2.1",
|
|
"numpy",
|
|
"scipy",
|
|
],
|
|
extras_require={
|
|
"vllm": ["vllm>=0.16"],
|
|
"triton": ["triton>=3.0"],
|
|
"test": ["pytest"],
|
|
},
|
|
classifiers=[
|
|
"Development Status :: 3 - Alpha",
|
|
"Intended Audience :: Science/Research",
|
|
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
|
"License :: OSI Approved :: MIT License",
|
|
"Programming Language :: Python :: 3",
|
|
],
|
|
)
|