mindspeed-mm-vlm

name: mindspeed-mm-vlm description: Universal VLM (vision-language understanding model) training guide for Huawei Ascend NPU using MindSpeed-MM. Covers all three framework patterns (Megatron, FSDP2, Custom trainers), weight conversion, dataset preparation (MLLM JSON format), fine-tuning, inference, and evaluation. Supports Qwen2.5VL, Qwen2VL, Qwen3VL, InternVL2.5/3/3.5, GLM4.1V, GLM4.5V, DeepSeekVL2, DeepSeekOCR, Ming, and more. Use when training or fine-tuning any multimodal understanding model on Ascend NPU. keywords: - mindspeed-mm - vlm - multimodal understanding - qwen2.5vl - qwen3vl - internvl - glm4v - deepseekvl - finetune - fine-tuning - vision-language - megatron - fsdp2

MindSpeed-MM VLM (Vision-Language Model) Training

This Skill guides users through training multimodal understanding (VLM) models on Huawei Ascend NPU using MindSpeed-MM. It uses Qwen2.5VL-3B as the flagship example and covers the end-to-end fine-tuning workflow.

Prerequisites

Critical: For most VLMs (Qwen2.5VL, Qwen2VL, InternVL, GLM4V, DeepSeekVL2), follow the manual install flow below. Do NOT use bash scripts/install.sh — official MindSpeed-MM docs state it only fully supports Qwen3/Qwen3.5. (For Qwen3VL / Qwen3.5, use one-click install + bash examples/qwen3_5/install_extensions.sh.)

Step P1: Clone repositories

git clone https://gitcode.com/Ascend/MindSpeed-MM.git /root/workspace/MindSpeed-MM
git clone https://github.com/NVIDIA/Megatron-LM.git /root/workspace/Megatron-LM
cd /root/workspace/Megatron-LM && git checkout core_v0.12.1
cp -r megatron /root/workspace/MindSpeed-MM/
cd /root/workspace/MindSpeed-MM

Step P2: Install PyTorch + torch_npu

Always use pre-built wheels from Ascend PyTorch Releases. Do not use pip install torch==2.7.1 — aarch64 wheels on PyPI are unreliable.

# Download matching wheels from https://gitcode.com/Ascend/pytorch/releases
# Replace cp310 with cp311 if using Python 3.11
pip install torch-2.7.1-cp310-cp310-manylinux_2_28_aarch64.whl
pip install torch_npu-2.7.1rc1-cp310-cp310-manylinux_2_28_aarch64.whl
# For x86_64: same naming pattern with _x86_64 suffix

# Required by torch_npu and other components:
pip install numpy pyyaml scipy attrs decorator psutil

Step P3: Install MindSpeed

git clone https://gitcode.com/Ascend/MindSpeed.git /root/workspace/MindSpeed
cd /root/workspace/MindSpeed
git checkout 93c45456c7044bacddebc5072316c01006c938f9
pip install -r requirements.txt
pip install -e .
cd /root/workspace/MindSpeed-MM

Step P4: Install MindSpeed-MM base dependencies

# Installs base stack from pyproject.toml: transformers==4.57.0, diffusers==0.30.3, peft==0.7.1, etc.
pip install -e .

Step P5: Activate CANN environment

source /usr/local/Ascend/cann/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

Step P6: Smoke test

python3 -c "
import torch, torch_npu
assert torch.npu.is_available(), 'NPU not available'
print(f'NPU count: {torch_npu.npu.device_count()}')
import transformers, diffusers, mindspeed
print(f'transformers={transformers.__version__}, diffusers={diffusers.__version__}')
"

Then proceed to Step 0 below for model-specific dependencies. For Qwen2.5VL the base is sufficient; other models need overlays (see Step 0 table).

For detailed troubleshooting and alternative install paths, see mindspeed-mm-env-setup.

Supported VLM Models

Entry Script Note: VLM models use different entry scripts. Check the shell script in examples/<model_name>/ for the exact command — do not assume from the model name.

How to Train Any VLM Model

The workflow for training any VLM model in MindSpeed-MM follows a universal pattern:

1. Find the example: Look in examples/<model_name>/ for available shell scripts (finetune, pretrain, inference, evaluate)
2. Read the shell script: Identify the entry script, config files, and parallelism settings
3. Check weight conversion: Determine if the model needs mm-convert (Megatron models), DCP conversion (FSDP2 models), or uses HF weights directly (custom trainer models)
4. Modify config files: Update data paths and weight paths in the config files (JSON or YAML depending on the framework)
5. Launch the shell script: bash examples/<model_name>/finetune_<model>_<size>.sh

For other models, adapt the Qwen2.5VL quick start below. Use the Model Registry to look up the exact entry script, converter, and backend for your target model.

Framework Patterns

All VLM models in MindSpeed-MM follow one of three framework patterns:

Pattern 1: Megatron (Qwen2.5VL, Qwen2VL, InternVL2.5, InternVL3, GLM4.1V, DeepSeekVL2)

# Uses JSON config files: data.json + model.json
# Weight conversion: mm-convert <Converter> hf_to_mm
torchrun $DISTRIBUTED_ARGS pretrain_vlm.py \
  --mm-data examples/<model>/data.json \
  --mm-model examples/<model>/model.json \
  --load $MM_WEIGHT_PATH ...

Pattern 2: FSDP2 (Qwen3VL, InternVL3.5, GLM4.5V)

# Uses YAML config file; requires CUDA_DEVICE_MAX_CONNECTIONS=2
# Weight conversion: mm-convert <Converter> hf_to_dcp
# MoE models may need: --init-model-with-meta-device
CUDA_DEVICE_MAX_CONNECTIONS=2 torchrun $DISTRIBUTED_ARGS pretrain_transformers.py \
  --fsdp2-config-path examples/<model>/fsdp2_config.yaml ...

Pattern 3: Custom Trainers (DeepSeekOCR, DeepSeekOCR2, Ming)

# Standalone entry scripts; uses HF weights directly (no conversion needed)
torchrun $DISTRIBUTED_ARGS examples/<model>/finetune_<model>.py <args>

Quick Start: Qwen2.5VL-3B Fine-Tuning (End-to-End)

This is the flagship Megatron-pattern example. For other models, follow the same workflow structure but use configs and scripts from examples/<model_name>/. FSDP2 models use YAML configs instead of JSON; Custom trainer models skip weight conversion entirely. See the "Framework Patterns" section above.

Step 0: Model-Specific Dependencies

Different VLM models have different version requirements for libraries such as transformers. Some of these requirements conflict, so environment isolation is necessary.

Important: Qwen3VL and DeepSeekOCR have conflicting transformers version requirements. If you need to support both, use separate Docker containers or conda environments. See per-model-deps.md for details.

Step 1: Weight Download and Conversion

1.1 Download HuggingFace Weights

# Download from HuggingFace (requires proxy)
http_proxy=http://127.0.0.1:58232 https_proxy=http://127.0.0.1:58232 \
  huggingface-cli download Qwen/Qwen2.5-VL-3B-Instruct \
  --local-dir ckpt/hf_path/Qwen2.5-VL-3B-Instruct

1.2 HF to MM Format Conversion

MindSpeed-MM uses its own weight format (MM format). Weights must be converted before training.

mm-convert Qwen2_5_VLConverter hf_to_mm \
  --cfg.mm_dir "ckpt/mm_path/Qwen2.5-VL-3B-Instruct" \
  --cfg.hf_config.hf_dir "ckpt/hf_path/Qwen2.5-VL-3B-Instruct" \
  --cfg.parallel_config.llm_pp_layers [[36]] \
  --cfg.parallel_config.vit_pp_layers [[32]] \
  --cfg.parallel_config.tp_size 1

Key parameter descriptions:

Important: llm_pp_layers and vit_pp_layers must match the pipeline_num_layers in model.json. When PP > 1, you need to split the layer counts, e.g., [[18,18]] for PP=2.

1.3 Converters for Different Models

Step 2: Dataset Preparation

VLM training uses the MLLM JSON format. This example uses COCO2017 + LLaVA-Instruct-150K.

2.1 Download Data

# 1. Download COCO2017 training images
mkdir -p data/COCO2017/train2017
# Extract to data/COCO2017/train2017/

# 2. Download LLaVA-Instruct-150K
# Download llava_instruct_150k.json to data/

2.2 Convert to MLLM Format

Important: The conversion script uses hardcoded relative paths. It expects ./data/llava_instruct_150k.json as input and ./data/COCO2017/train2017/ for image lookup. Run it from the MindSpeed-MM root directory, and ensure data is at ./data/ (or create symlinks).

cd /root/workspace/MindSpeed-MM  # Must run from repo root
python examples/qwen2vl/llava_instruct_2_mllm_demo_format.py

Output: data/mllm_format_llava_instruct_data.json

The script converts LLaVA format (conversations/image) to MLLM format (messages/images), which is what MindSpeed-MM expects.

2.3 Data Directory Structure

data/
├── COCO2017/train2017/          # Image files
│   ├── 000000000009.jpg
│   ├── 000000000025.jpg
│   └── ...
├── llava_instruct_150k.json     # Original data (for conversion)
└── mllm_format_llava_instruct_data.json  # MLLM format (for training)

Field name matching: The field names in your data JSON must match the attr section in data.json config. Default mapping: messages for conversations, images for image paths. Using different field names (e.g., conversations instead of messages) will cause KeyError.

For detailed MLLM JSON format specifications, see data-format.md.

Step 3: Configuration Files

Qwen2.5VL-3B fine-tuning requires two JSON configuration files:

3.1 data.json (Data Configuration)

Path: examples/qwen2.5vl/data_3b.json

{
    "dataset_param": {
        "dataset_type": "huggingface",
        "preprocess_parameters": {
            "model_name_or_path": "./ckpt/hf_path/Qwen2.5-VL-3B-Instruct"
        },
        "basic_parameters": {
            "dataset_dir": "./data",
            "dataset": "./data/mllm_format_llava_instruct_data.json",
            "cache_dir": "./data/cache_dir"
        }
    }
}

Must edit before running — update these 3 paths to match your environment:

Multi-node Note: cache_dir stores preprocessing cache. During multi-node training, each node must use a local path and must not point to an NFS shared directory, otherwise concurrent write conflicts will occur.

3.2 model.json (Model Configuration)

Path: examples/qwen2.5vl/model_3b.json

model.json defines three components of the model architecture:

Important: The pipeline_num_layers field defines the PP partitioning scheme and must exactly match the llm_pp_layers / vit_pp_layers used during weight conversion.

Step 4: Training Script Configuration

The training launch script is located at examples/qwen2.5vl/finetune_qwen2_5_vl_3b.sh.

4.1 Key Variables

LOAD_PATH="ckpt/mm_path/Qwen2.5-VL-3B-Instruct"   # Converted MM weights
SAVE_PATH="save_dir"                                 # Training output save path
DATA_CONFIG="examples/qwen2.5vl/data_3b.json"        # Data configuration
MODEL_CONFIG="examples/qwen2.5vl/model_3b.json"      # Model configuration
TP=1; PP=1; CP=1                                      # Parallelism configuration
MBS=1; GRAD_ACC_STEP=32                               # Batch size configuration

4.2 Important Training Parameters

4.3 Docker Users

Docker users: If your container lacks --ipc=host or --shm-size=16g, set --num-workers 0 in the training script to avoid Bus error from DataLoader workers.

4.4 Launch Command

torchrun $DISTRIBUTED_ARGS pretrain_vlm.py $GPT_ARGS $MM_ARGS $OUTPUT_ARGS

Note: Qwen2.5VL uses pretrain_qwen2vl.py or pretrain_vlm.py as the entry point. Different versions of the script may vary; refer to the actual script in your installation.

4.5 Single-Node vs Multi-Node

# Single node, 8 NPUs
MASTER_ADDR=localhost
NNODES=1
NODE_RANK=0

# Multi-node (2 nodes)
# Node 0 (master node)
MASTER_ADDR=192.168.1.100  # Master node IP
NNODES=2
NODE_RANK=0

# Node 1
MASTER_ADDR=192.168.1.100  # Same as above; points to the master node
NNODES=2
NODE_RANK=1

Step 5: Launch Training

bash examples/qwen2.5vl/finetune_qwen2_5_vl_3b.sh

Key metrics to monitor in training logs:

• lm loss: Language model loss; should decrease steadily
• grad norm: Gradient norm; excessively large values may indicate the need to adjust the learning rate
• iteration time: Time per training iteration

Step 6: Inference Verification (Optional)

After training, you can run inference verification. Note: no 3B-specific inference script exists; use the 7B script and adjust the model/config paths:

# Edit inference_qwen2_5_vl_7b.sh to point to your 3B checkpoint, then:
bash examples/qwen2.5vl/inference_qwen2_5_vl_7b.sh

Inference requires an inference JSON configuration (inference_qwen2_5_vl_7b.json) specifying the model path and test images. The inference script uses the inference_vlm.py entry point.

Step 7: Evaluation (Optional)

# Similarly, use the 7B evaluation script with 3B paths:
bash examples/qwen2.5vl/evaluate_qwen2_5_vl_7b.sh

Note: Only 7B and 72B inference/evaluation scripts are provided. For 3B or 32B, copy and adapt the 7B scripts with the appropriate model config and checkpoint paths.

Recommended Parallelism Configurations by Model Size

Post-Training: DPO Preference Alignment

Qwen2.5VL supports DPO post-training using the posttrain_qwen2vl_dpo.py entry point:

bash examples/qwen2vl/finetune_qwen2vl_72b_dpo.sh

See the relevant files in the examples/qwen2vl/ directory for DPO configuration and data format details.

FAQ

Q: Training fails with shape mismatch after weight conversion

pipeline_num_layers does not match the llm_pp_layers / vit_pp_layers used during weight conversion. Check that the layer count configuration in model.json matches the parameters passed to the mm-convert command.

Q: Data loading hangs during multi-node training

cache_dir is set to an NFS shared mount path. Change it to a local path (e.g., /tmp/cache_dir) so each node preprocesses data independently.

Q: transformers version conflict

Different VLM models require different versions of transformers. Qwen3VL needs the latest version, while DeepSeekOCR needs 4.46.3. Use separate containers to isolate environments.

Q: Images fail to load during inference

Check that the image paths in the inference JSON are correct and that the image files exist in a supported format (JPEG/PNG).

Q: Training OOM (out of memory)

Reduce --micro-batch-size to 1, enable --use-distributed-optimizer, or increase TP/PP parallelism.

Q: Training fails with Communication_Error_Bind_IP_Port

A stale process from a previous run is holding the port. Fix:

ps aux | grep torchrun | grep -v grep | awk '{print $2}' | xargs kill -9
# Or change MASTER_PORT in the training script

Related Skills

• mindspeed-mm-env-setup - MindSpeed-MM Environment Setup
• ascend-docker - Ascend Docker Container Configuration
• hccl-test - Multi-Device Communication Testing

Reference Resources

• Model Delta Cards - Execution checklists for non-flagship models (Qwen3VL, InternVL3.5, GLM4.5V, DeepSeekVL2, DeepSeekOCR, Ming)
• Model Dependency Configuration - Dependencies and version requirements for each VLM model
• Data Format Specification - Detailed MLLM JSON format documentation
• Model Registry - Complete lookup table for all models
• MindSpeed-MM Repository