vllm-omni-quantization

name: vllm-omni-quantization description: Use when working on vLLM-Omni quantization for autoregressive, diffusion, or multi-stage omni models, choosing methods such as awq, gptq, fp8, int8, gguf, or ModelOpt checkpoints, adding quantized model support, or debugging memory, loader, quality, or performance issues.

vLLM-Omni Quantization

Overview

Use this skill for vllm-omni quantization work. The current codebase has a unified quantization framework centered on vllm_omni.quantization.build_quant_config(), but the runtime integration still splits into distinct patterns:

• AR and general quantization inherited from upstream vllm
• diffusion quantization for DiT models in vllm-omni, currently fp8, int8, and gguf
• multi-stage omni quantization using scoped pre-quantized checkpoints such as Qwen3-Omni thinker ModelOpt checkpoints

Core principle: keep generic quantization infrastructure in upstream vllm. Keep vllm-omni focused on unified config routing, component scoping, diffusion-specific model wiring, adapter logic, and verification.

Quick Decision

When to Use

• Choosing a quantization method for memory or throughput
• Checking whether a modality or model family actually supports quantization
• Using the unified build_quant_config() entrypoint or per-component config dicts
• Enabling diffusion fp8, int8, or gguf
• Adding a new diffusion quantization method or a pre-quantized multi-stage model path
• Debugging quantized loading, tensor-name mapping, shape mismatch, quality drift, or performance regressions

AR vs Diffusion Boundary

• AR and general quantization usually mean upstream vllm methods such as awq, gptq, fp8, and KV-cache FP8.
• Diffusion quantization means vllm-omni DiT-specific integration on top of the unified framework and should not duplicate upstream vllm kernels or config semantics.
• Multi-stage omni quantization often means pre-quantized checkpoints whose scope must be constrained to the intended component, such as the thinker language_model.

Rule: if a new method is missing generic kernels, loader behavior, or config classes, fix upstream vllm first. vllm-omni should add thin wrappers, component routing, and model-specific wiring, not a private quantization stack.

Example: Enable FP8 for a Diffusion Model

# 1. Start server with fp8 quantization
vllm serve black-forest-labs/FLUX.1-dev --omni \
  --quantization fp8 --tensor-parallel-size 2

# 2. Verify quantized model loaded correctly
curl -s http://localhost:8091/v1/models | python3 -c "
import sys, json
models = json.load(sys.stdin)['data']
print(f'Loaded: {models[0][\"id\"]}') if models else print('ERROR: No models loaded')
"

# 3. Generate test image with fixed seed for quality comparison
curl -s http://localhost:8091/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"a cat","seed":42}' -o test_fp8.json

# 4. Compare against BF16 baseline (run separately without --quantization)
# If PSNR < 25 dB or visual artifacts appear, check references/diffusion.md

Common Mistakes

References

• AR and general methods: references/methods.md
• Model and modality support matrix: references/modality-compat.md
• Diffusion fp8, int8, gguf, and unified-framework workflow: references/diffusion.md
• Adding quantization support to a new model: references/adding-models.md