Lsglang GPU + NUMA Dual Parallel [中文]

Lsglang is a special extension of sglang that fully utilizes CPU and GPU computing resources with an efficient GPU parallel + NUMA parallel architecture, suitable for MOE model hybrid inference.

Core Engine: The actual hybrid inference functionality—including CPU-GPU collaborative computation, NUMA-aware scheduling, expert weight management, and quantization kernel execution—is powered entirely by lk_moe, a highly optimized MOE hybrid inference engine. Within LvLLM (for vLLM) and Lsglang (for sglang), each MOE layer can flexibly choose between the original GPU computation path or invoke lk_moe for hybrid inference. For DeepSeek V4, specialized versions are also available: Lvllmds4 (SM120+) and Lvllmds4-x (SM80+).

System Features

• GPU + NUMA Dual Parallel: Supports three computing modes: CPU-GPU hybrid decoding, CPU-GPU hybrid prefill, and GPU prefill
• VRAM + Memory Load Balancing: Total model footprint = VRAM + memory, accommodating model 1+1=2, 100% VRAM utilization ^{Note 1}
• GPU Prefill Optimization: GPU prefill runs in parallel with CPU-GPU hybrid decoding, achieving nearly 100% GPU utilization
• NUMA Thread Optimization: Cross-node communication as low as 3%, L3 cache hit rate over 50%, GPU load can reach 33% to 50% during decoding

Relationship with sglang

Lsglang uses the latest sglang source code and has redesigned and implemented the MOE model hybrid inference module, maintaining 100% full compatibility with sglang^{Note 1}.

Note 1: x86 CPUs with AVX2+ instruction sets and Nvidia GPUs with sm80+ architectures

Usage Guide [中文]

• Performance Benchmark
• Version Changes
• Supported Models
• Supported Quantization Formats
• Running Command Reference
• Configuration Parameters
• Installation Steps
• Optimization

Performance Benchmark

Open GPU Prefill, max_num_batched_tokens=8192 (Row 1), max_num_batched_tokens=32768 (Row 2) | Model | Version | CPU | Memory | GPU | Prefill | Decode | Speculative Decoding | |-------|---------|-----|--------|-----|---------|--------|---------| | deepseek-ai/DeepSeek-V4-Flash-0731 | Lsglang-v1.4.7 | EPYC 7642 2 | 16 channels ddr4 3200 | 5060Ti * 2 | 780 t/s [input 32768]| 25 t/s [input 32768]| 35~47 t/s | | deepseek-ai/DeepSeek-V4-Flash-0731 | Lsglang-v1.4.7 | EPYC 9684x 2 | 24 channels ddr5 4800 | pro 6000 * 1 | 4600 t/s [input 131072]| 75 t/s [input 131072]| 100~132 t/s |

Version Changes

2026-07-08: Lsglang-v1.4.1 - add ModelOpt W4A16 NVFP4 quantization types support, for example: nvidia/GLM-5.2-NVFP4
2026-07-05: Lsglang-v1.4.0 - Optimize GPU prefill speed, CPU AVX512 optimization, removed LVLLM_GPU_RESIDENT_MOE_EXPERTS, update to sglang v0.5.14
2026-06-05: Lsglang-v1.3.0 - Upgraded lk_moe module, supports nvfp4, mxfp4 quantization types, added LVLLM_GPU_RESIDENT_MOE_EXPERTS, removed LVLLM_MOE_USE_WEIGHT, LVLLM_MOE_QUANT_ON_GPU
2026-04-06: Lsglang-v1.2.0 - Enhanced energy saving effect with LK_POWER_SAVING=1, supports mixed MOE layer inference with FP8+BF16+AWQ4bit
2026-04-03: Lsglang-v1.1.4 - Supports local compilation of sgl-kernel to fix known issues
2026-03-11: Lsglang-v1.1.3 - FP8 and AWQ4bit models no longer occupy additional memory when GPU Prefill is enabled, FP8 models removed TO_DTYPE runtime type conversion, KEEP temporarily does not support GPU Prefill
                             Note 1: RTX 30 series GPUs can enable GPU Prefill for FP8 models by removing the LVLLM_GPU_RESIDENT_MOE_LAYERS parameter
2026-03-05: Lsglang-v1.1.0 - Supports GPU prefill, updated corresponding commands (FP8 models not supported on RTX 3090 and below architectures)
2026-02-25: Lsglang-v1.0.6 - Fixed known issues, added new model support  
2026-02-10: Lsglang-v1.0.0 -  Ported from LvLLM project [https://github.com/guqiong96/Lvllm], verified BF16, F16 original models, FP8 original models, AWQ 4bit symmetric quantization models.

Supported Models

Most original MOE models verified by Lsglang

Unlisted original MOE models from Qwen3 series, GLM series, and MiniMax series are theoretically supported and pending actual testing.

Supported Quantization Formats

Note 1: https://hf-mirror.com/cyankiwi provides AWQ 4bit symmetric quantization models

Running Command Reference

LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREAD_BINDING=CPU_CORE \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=2048 \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_RESIDENT_MOE_LAYERS=0-1,33-34 \
LVLLM_ENABLE_NUMA_INTERLEAVE=1 \
LVLLM_ENABLE_MOE_LAYERWISE_LOAD=1 \
python -m sglang.launch_server \
    --model /home/guqiong/Models/Qwen3.6-35B-A3B \
    --served-model-name Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8070 \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --max-running-requests 2 \
    --chunked-prefill-size 32000 \
    --max-total-tokens 66000 \
    --mem-fraction-static 0.90 \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --disable-shared-experts-fusion

GLM-5.2-NVFP4 [RTX PRO 6000 * 2]

CUDA_DEVICE_ORDER=PCI_BUS_ID \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_VISIBLE_DEVICES=1,0 \
SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 \
LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREAD_BINDING=CPU_CORE \
LK_THREADS=60 \
OMP_NUM_THREADS=60 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=512 \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_RESIDENT_MOE_LAYERS=0-18 \
LVLLM_ENABLE_NUMA_INTERLEAVE=1 \
python -m sglang.launch_server \
    --model /mnt/ktd/glm52 \
    --served-model-name GLM-5.2-NVFP4 \
    --host 0.0.0.0 \
    --port 8070 \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --max-running-requests 2 \
    --chunked-prefill-size 16384 \
    --max-total-tokens 66000 \
    --mem-fraction-static 0.95 \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --disable-shared-experts-fusion \
    -cuda-graph-backend-prefill disabled \
    --attention-backend triton

# or use flashinfer
    --attention-backend flashinfer

GLM-5.2-NVFP4 [RTX 3090 * 2]

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREAD_BINDING=CPU_CORE \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LVLLM_ENABLE_NUMA_INTERLEAVE=1 \
python -m sglang.launch_server \
    --model /home/guqiong/Models/GLM-5.2-NVFP4 \
    --served-model-name GLM-5.2-NVFP4 \
    --host 0.0.0.0 \
    --port 8070 \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --max-running-requests 2 \
    --chunked-prefill-size 256 \
    --max-total-tokens 8192 \
    --mem-fraction-static 0.98 \
    --kv-cache-dtype bfloat16 \
    --attention-backend triton \
    --moe-runner-backend marlin \
    --cuda-graph-backend-prefill disabled \
    --disable-shared-experts-fusion \
    --tool-call-parser glm47 \
    --reasoning-parser glm45

# or use flashinfer
    --attention-backend flashinfer

Configuration Parameters

Installation Steps

1. Install CUDA 13.2.1

# Uninstall old CUDA and NVIDIA driver
sudo /usr/local/cuda/bin/cuda-uninstaller   
sudo nvidia-uninstall

# Download and install CUDA 13.2.1 
wget https://developer.download.nvidia.com/compute/cuda/13.2.1/local_installers/cuda_13.2.1_595.58.03_linux.run
sudo sh cuda_13.2.1_595.58.03_linux.run

2. Create Python Environment

conda create -n Lsglang python==3.12.11
conda activate Lsglang

# Upgrade libstdcxx-ng (avoid glibcxx version issues)
conda install -c conda-forge libstdcxx-ng
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Install NUMA library
sudo apt-get install libnuma-dev      # Ubuntu
sudo dnf install numactl-devel        # Rocky Linux

3. Install Lsglang

pip install lsglang

Compile from Source and Install Lsglang

# 克隆仓库
git clone https://github.com/guqiong96/Lsglang.git
cd Lsglang
pip install -U setuptools wheel scikit-build-core cmake
pip install torchaudio triton torchvision torch==2.11.0
pip install grpcio-tools 
MAX_JOBS=32 NVCC_THREADS=1 CMAKE_BUILD_TYPE=Release  CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release" pip install -e "python" --no-build-isolation -vvv

Parameter Explanation: - MAX_JOBS=32 NVCC_THREADS=1: Reduce compilation memory usage - CMAKE_BUILD_TYPE=Release: Performance optimization option - CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release: Performance optimization option

Optimization

MoE Resident in VRAM, Linear Increase in Decode and Prefill Speed

# MoE layers 0-5 resident in VRAM
# Format 0,1,8-9 means MoE layers 0,1,8-9 resident in VRAM
# Some models start at non-zero layer numbers, e.g., Step-3.5-Flash starts at layer 3 
LVLLM_GPU_RESIDENT_MOE_LAYERS=0-5 

Enable GPU Prefill

# Prefetch 1 layer
LVLLM_GPU_PREFETCH_WINDOW=1
# GPU prefill starts when input length reaches 4096
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=4096 
#配合修改最大批处理大小
--chunked-prefill-size 32000 

Disable GPU Prefill

# Disable GPU prefill
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0
#配合修改最大批处理大小
--chunked-prefill-size 4096 

Thread Binding to CPU Cores

# Bind to CPU cores (including hyper-threading logical cores), best performance
LK_THREAD_BINDING=CPU_CORE 
# Bind to NUMA nodes, second best option, resolves extreme performance issues on virtualization platforms and multi-instance running
LK_THREAD_BINDING=NUMA_NODE 

BIOS NUMA Settings

AMD EPYC: Set NPS4 for best performance
Intel XEON: Set SNC4 for best performance
# Some virtualization platforms or Intel platforms should not set 5 or 10 nodes, set 2 nodes to avoid performance issues
Generally: 2, 4, 8 nodes, supports up to 32 nodes, more nodes is better, node count as multiple of GPU count for best performance 

Thread Count Settings

# Hyper-Threading enabled: total physical cores ÷ number of GPUs
# Hyper-Threading disabled: (total physical cores - 2) ÷ number of GPUs
# 96 cores, 2 GPUs → 48 threads per GPU
LK_THREADS=48
# Total threads exceeding physical core count may cause performance issues    

VRAM Settings

# Maximum batch size occupies significant VRAM, adjust accordingly
--chunked-prefill-size 32000  

CPU Power Saving

# When enabled, reduces CPU temperature during inference with slight performance decrease
LK_POWER_SAVING=1