eagle3-triage

name: eagle3-triage description: > Triage a failed EAGLE3 pipeline run. Identifies which step failed (data synthesis, hidden state dump, training, or benchmark), diagnoses root cause from logs, and suggests fixes. Use when user reports an EAGLE3 pipeline failure or asks why a specific step failed. Also helps debug new model support issues. user_invocable: true

EAGLE3 Pipeline Triage

Diagnose failures in the 4-step EAGLE3 offline pipeline. This skill walks through each step, identifies the failure point, and provides actionable fixes.

Pipeline Overview

Step 0 — Locate the experiment

Ask the user for one of:

• Experiment directory (e.g., the --job-dir passed to launch.py or slurm.py)
• The model name / YAML they ran

Find recent experiments under the job directory:

ls -td experiments/cicd/cicd_* | head -10
# or wherever --job-dir was pointed

Each experiment directory contains one subdirectory per task (task_0 through task_3), each with a log file whose name varies by launch mode (Slurm: sbatch_*.out, local Docker: *.log).

Step 1 — Fetch logs for the failed task

Match the log files generally and read the tail of each — errors appear at the end:

find experiments/<exp_id>/ -type f \( -name '*.out' -o -name '*.log' \) | sort | while read -r f; do
  echo "=== $f ==="; tail -200 "$f"; echo
done

Look for the first task with a non-zero exit code or error message.

Step 2 — Diagnose by step

task_0 failures (Data Synthesis)

How it works: Launches a vLLM OpenAI-compatible server, polls /health until ready, then runs query.py to generate synthetic prompt/response pairs. Output goes to /scratchspace/data/.

task_1 failures (Hidden State Dump)

How it works: Loads the target model and runs a forward pass on each conversation, saving hidden states as .pt files in /scratchspace/offline_hidden_states/.

Three backends are available:

task_2 failures (Training)

How it works: Installs requirements, runs launch_train.sh (Accelerate + FSDP) with the config from modelopt_recipes/general/speculative_decoding/eagle3.yaml, then exports via export_hf_checkpoint.py. Output: /scratchspace/eagle3/ and /scratchspace/export/.

task_3 failures (Benchmark)

How it works: Launches vLLM with the target + draft model, runs acceptance rate and throughput benchmarks. Output: JSON files.

Step 3 — Check for new-model-specific issues

If the user is adding support for a new model, also check:

1. Is the model a VLM? → Use dump_offline_data_hf.sh (text-only path, no vision encoder invoked)
2. Does the model use sliding window attention (SWA)? → TRT-LLM backend won't work; use HF or vLLM
3. Does the model need trust_remote_code? → Add to task_0 args AND task_3 args
4. Is the model MoE? → Check eagle_config.json intermediate_size matches model's moe_intermediate_size
5. Is the model architecture recognized by EAGLE3 training? → may need code changes in modelopt/torch/speculative/
6. Custom tokenizer? → May need additional environment vars (e.g., TIKTOKEN_RS_CACHE_DIR)

Step 4 — Suggest fix and next steps

After diagnosis, provide:

1. Root cause — one-line summary
2. Fix — specific config change, code edit, or command to run
3. How to re-run — skip earlier successful steps by pointing to existing scratchspace artifacts

To skip task_0 and task_1 and re-run from task_2:

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_1.skip=true \
    --yes

To run only task_1 standalone (using existing task_0 data):

uv run launch.py --yaml examples/<Org>/<Model>/hf_offline_eagle3.yaml \
    pipeline.task_0.skip=true \
    pipeline.task_2.skip=true \
    pipeline.task_3.skip=true \
    --yes

If the fix requires code changes in ModelOpt (e.g., supporting a new model architecture), note that a separate PR in the modelopt repo is needed.

Step 5 — Record the failure pattern

If you encounter a failure pattern not seen before, capture it in the team's internal triage tracker — the symptom, root cause, and fix — so the next engineer debugging the same issue benefits.