1Cat-vLLM

一猫之下始终相信，V100 不该在今天的大模型浪潮中被轻易宣判“过时”。

1Cat-vLLM 是一个专注于 SM70 / Tesla V100 的 vLLM 工程分支。我们 不追求覆盖所有硬件，而是围绕 Volta 架构补齐并优化现代大模型推理的 关键路径，包括量化推理、注意力后端、长上下文、MTP 投机解码、 CUDA Graph、分布式通信、运行时策略和部署工具链，让 Qwen、DeepSeek 等新模型在 V100 上从“能跑”走向“可部署、可验证、可持续优化”。

我们希望把来自真实 V100 环境的工程实现、性能数据和验证过程贡献给 开源社区，也欢迎仍在使用 V100 的个人开发者、工作室和团队参与测试、 复现、反馈和共建，让这批依然有价值的算力继续发挥作用。

1Cat-vLLM is a vLLM engineering fork focused on SM70 / Tesla V100. Rather than targeting every hardware platform, it fills and optimizes the critical inference paths required by modern models on Volta, including quantized inference, attention backends, long-context serving, MTP speculative decoding, CUDA Graph execution, distributed communication, runtime policies, and deployment tooling.

The goal is to make models such as Qwen and DeepSeek not merely runnable on V100, but deployable, reproducible, and continuously optimizable.

Project Focus

• V100 / SM70 first: optimized for Tesla V100 rather than being a generic multi-hardware fork.
• AWQ on Volta: AWQ 4-bit inference paths for dense and MoE Qwen models on SM70.
• V100 FlashAttention path: FLASH_ATTN_V100 decode and prefill backend for Volta GPUs, with SM70 compile-graph, guarded XQA decode, and D=256 paged-prefix low-smem fast paths enabled by default.
• DeepSeek V4 Flash: supports running the original, unmodified DeepSeek V4 Flash weights across eight GPUs.
• Day-0 Qwen3.8-27B support: native Day-0 support for Qwen3.8-27B.
• Quantization support: FP8, NVFP4, MXFP4, AWQ, and GPTQ inference paths are included for supported model and hardware combinations. Availability and production status vary by checkpoint and runtime route.
• Long-context serving: public profiles default to 256K context where the model and memory budget allow it.
• MTP serving: Qwen3.6-class MTP speculative decoding remains available as an explicit opt-in path; long-context public profiles default to no MTP.
• Image inputs by default: SM70 FLASH_ATTN_V100 profiles allow one image per prompt by default; video inputs remain opt-in.
• Tool calling and OpenAI API compatibility: validated with OpenAI-style clients such as Cherry Studio, OpenClaw, and similar tools.
• Experimental FP8 work: FP8 model and KV-cache paths are included for validation, but they are not production defaults.
• Experimental DFlash work: included for continued research and validation.

Recommended Model Providers

• tclf90/Qwen3.6-27B-AWQ
• tclf90/Qwen3.6-35B-A3B-AWQ
• tclf90/Qwen3.5-122B-A10B-AWQ for larger 4-GPU setups
• Qwen/Qwen3.8-27B for the latest supported Qwen release

The launch examples use local paths such as /path/to/Qwen3.6-27B-AWQ. Replace them with your local model path or a Hugging Face repository id.

Hardware Target

The public commands are written for V100 Qwen serving workloads. Image inputs are enabled by default on the SM70 FLASH_ATTN_V100 path; video inputs are disabled by default and should be enabled explicitly only after local memory validation.

Typical model placement:

• Qwen3.6-27B-AWQ: TP1/TP2/TP4 supported; TP4 is the public reference.
• Qwen3.6-27B-AWQ + MTP: explicit opt-in profile for local validation, not the long-context public default.
• Qwen3.6-35B-A3B-AWQ: TP4 recommended.
• Qwen3.5-122B-A10B-AWQ: TP4 supported for larger deployments.
• DeepSeek-V4-Flash: the original mixed MXFP4/FP8 checkpoint is supported on eight 32 GB V100 GPUs with TP8.
• Qwen3.8-27B: supported natively from Day 0, with validated FP16 and FP8 serving paths.

Multimodal defaults:

• Default SM70 FLASH_ATTN_V100 serving allows image=1, video=0 when --limit-mm-per-prompt is not set.
• For text-only serving, pass --limit-mm-per-prompt '{"image":0,"video":0}' or use --language-model-only.
• For video workloads, pass an explicit limit such as --limit-mm-per-prompt '{"image":1,"video":1}' and retune memory settings.

Validated Stack

The public wheel path is validated on:

• OS: Ubuntu 24.04 LTS
• Python: 3.12
• CUDA toolkit: 12.8
• PyTorch: CUDA 12.8 runtime wheels
• GPU: Tesla V100 32 GB

Quick Start

1. Install CUDA 12.8

Use the official NVIDIA repository on Ubuntu 24.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-8

If the machine also has another CUDA toolkit installed, force build-time and runtime CUDA to 12.8:

export CUDA_HOME=/usr/local/cuda-12.8
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}
hash -r
nvcc -V

2. Create the Python environment

source /path/to/miniconda3/etc/profile.d/conda.sh
conda create -y -n 1cat-vllm-sm70 python=3.12
conda activate 1cat-vllm-sm70

python -m pip install --upgrade pip setuptools wheel

3. Install from Prebuilt Wheels

Prebuilt wheels are the recommended installation path for public users. Source builds are intended for kernel development.

Download the latest wheel assets from:

https://github.com/1CatAI/1Cat-vLLM/releases/latest

Install the wheel from the directory where you downloaded it:

python -m pip install --prefer-binary --no-cache-dir \
  --extra-index-url https://download.pytorch.org/whl/cu128 \
  ./1cat_vllm-*.whl

Notes:

• The 1cat_vllm wheel already bundles the flash_attn_v100 Python package and SM70 CUDA extensions.
• Runtime installation from wheels does not require the bundled lmdeploy source tree.
• Use Python 3.12 and CUDA 12.8.
• If your shell has a broken local proxy configured, unset it before installing: env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u all_proxy ....
• After installing from wheels, run python -m vllm... from a directory outside this source checkout, such as cd ~ or cd /tmp. Running inside the cloned repository makes Python import the local source tree instead of the wheel-installed CUDA extensions.

4. Verify the Environment

python - <<'PY'
import torch, triton, vllm, sys
import flash_attn_v100
from flash_attn_v100 import flash_attn_v100_cuda, paged_kv_utils
print("python", sys.version.split()[0])
print("torch", torch.__version__)
print("torch_cuda", torch.version.cuda)
print("triton", triton.__version__)
print("vllm", vllm.__version__)
print("flash_attn_v100", flash_attn_v100.__version__)
PY

Recommended Launch Commands

These are the recommended public serving commands for the 27B AWQ and 35B AWQ V100 profiles. When using prebuilt wheels, run them outside the source checkout so Python loads the installed package and its CUDA extensions.

Use CUDA_VISIBLE_DEVICES=0,1,2,3 only when you need to select a specific four-card V100 set.

Qwen3.6-27B-AWQ, TP4

python -m vllm.entrypoints.openai.api_server \
  --model /path/to/Qwen3.6-27B-AWQ \
  --served-model-name qwen3.6-27b-awq \
  --trust-remote-code \
  --attention-backend FLASH_ATTN_V100 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --host 0.0.0.0 \
  --port 8000

Qwen3.6-35B-A3B-AWQ, TP4

python -m vllm.entrypoints.openai.api_server \
  --model /path/to/Qwen3.6-35B-A3B-AWQ \
  --served-model-name qwen3.6-35b-a3b-awq \
  --trust-remote-code \
  --attention-backend FLASH_ATTN_V100 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --host 0.0.0.0 \
  --port 8000

OpenAI-Compatible Request Example

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer EMPTY' \
  -d '{
    "model": "qwen3.6-27b-awq",
    "messages": [{"role": "user", "content": "用一句话回答，2+2等于几？"}],
    "temperature": 0,
    "max_completion_tokens": 32,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

If the response is coherent and short, the API path is basically healthy.

Experimental Features

FP8

FP8 support is included for validation and research. It is not the stable public default.

• FP8 model execution on V100 is experimental.
• fp8_e5m2 KV cache can be used experimentally on V100.
• fp8_e4m3 is not the recommended V100 option in the current path.
• Do not add --calculate-kv-scales unless you are specifically testing KV scale calculation behavior.

Example:

--kv-cache-dtype fp8_e5m2

DFlash

DFlash is included as an experimental path for continued validation. Treat it as a research feature until you have validated speed and output quality on your own workload.

MTP

MTP is not enabled by default in the V100 public serving profile. Long-context decode on V100 can slow down significantly when MTP is enabled, so keep the default no-MTP path for 128K/256K style serving unless your own workload proves otherwise.

To explicitly test the previous automatic SM70 MTP4 profile:

export VLLM_1CAT_ENABLE_SM70_MTP_DEFAULTS=1

You can also pass an explicit --speculative-config when you want full control over speculative decoding settings.

Dense F16 Fast Path

VLLM_SM70_ENABLE_DENSE_F16_FASTPATH=1 is intended for targeted experiments. Keep it disabled for public MoE serving profiles unless you are explicitly benchmarking that path.

Source Build

Source build is supported, but it is not recommended for normal runtime deployment. Install the release wheels first unless you are changing CUDA, C++, or Triton code.

This repository includes the validated lmdeploy source tree under csrc/sm70_turbomind/lmdeploy, which is needed by the SM70 AWQ build path.

cd /path/to/1Cat-vLLM/vllm
test -d csrc/sm70_turbomind/lmdeploy

Install build dependencies:

source /path/to/miniconda3/etc/profile.d/conda.sh
conda activate 1cat-vllm-sm70

python -m pip install -r requirements/build/cuda.txt
python -m pip install -r requirements/cuda.txt
python -m pip install -r requirements/common.txt
python -m pip install cmake build

Build wheels:

export CUDA_HOME=/usr/local/cuda-12.8
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}
export TORCH_CUDA_ARCH_LIST="7.0;8.0"
export FLASH_ATTN_V100_CUDA_ARCH_LIST="7.0"
export MAX_JOBS=12
export NVCC_THREADS=1

rm -rf build vllm.egg-info
rm -rf .deps/*-build .deps/*-subbuild

pushd flash-attention-v100
python -m build --wheel --no-isolation --outdir ../dist-cu128-sm70
popd

python -m build --wheel --no-isolation --outdir dist-cu128-sm70

For editable development:

python -m pip install -e . --no-build-isolation

Benchmarking Notes

• First-request warmup is slow on V100 and should not be included in steady-state throughput.
• Browser-side OpenAI streaming throughput includes request overhead and should not be compared directly with strict incremental decode TPS.
• Long-context throughput depends strongly on TP, max_num_seqs, max_num_batched_tokens, prompt shape, and attention backend.
• If you publish a baseline, include the full launch command, GPU model, driver, CUDA runtime, model checkpoint, sampling parameters, prompt length, and decode length.

WeChat Community

群聊： 1Cat-vLLM 开源交流群

请使用微信扫描下方二维码加入群组：

提示：微信群二维码通常 7 天内有效。若扫描失败或提示过期，请联系微信号：YM_isi。

Roadmap

1Cat-vLLM will continue to prioritize Tesla V100 / SM70, making modern LLM inference more usable, stable, and deployable on existing GPUs.

• Near term: Complete Volta compatibility and validation for DeepSeek V4, GLM5, and other new model families, while improving long-context serving, quantization, multi-GPU parallelism, and runtime stability.
• Mid term: Expand the Volta model support matrix with a focus on models up to 300B parameters, backed by reproducible performance, quality, and release validation.
• Long term: Build on the Volta foundation and extend hardware support in stages: Volta → Turing/Ampere → CDNA, bringing sustainable LLM inference to a broader range of existing accelerators.

Repository Notes

• This fork focuses on SM70 quantized inference, V100-oriented attention and long-context tuning, model-specific runtime and deployment paths, and continued MTP and DFlash research.
• Production status is route-specific. Use the documented public profiles and validated release matrices as the source of truth.

Acknowledgements

• vLLM
• lmdeploy / TurboMind
• flash-attention-v100
• marlin_v100

License

This repository follows the upstream vLLM license model. See LICENSE.