nightlydeploy - SKILL.md Agent Skill

name: nightly:deploy description: Diagnose deployment failures in nightly E2E runs (helmfile, helm, install.sh)

Nightly Deploy Failure Analysis

Diagnose failures during the deployment phase of nightly E2E runs.

Context-Safe Execution (MANDATORY)

export LOG_DIR=/tmp/llm-d/nightly-rca
mkdir -p $LOG_DIR

Failed Step Patterns

Step	Workflow Type	Deploy Method
`Deploy guide via helmfile`	helmfile-based	`helmfile apply`
`Deploy guide via custom script`	kustomize+helm	Custom bash script
`Deploy infrastructure`	WVA (legacy)	`deploy/install.sh`
`Install gateway provider`	OCP helmfile	`istioctl install` or helm

Diagnosis

1. Download Artifacts

RUN_ID=<run-id>
REPO=llm-d/llm-d

# Download pod logs artifact (collected on every run, even failures)
unset GITHUB_TOKEN && gh run download "$RUN_ID" --repo "$REPO" \
  -n "nightly-pod-logs-<guide-name>-<platform>" \
  -D $LOG_DIR/pod-logs-$RUN_ID/ 2>/dev/null

# Download helmfile deploy log
unset GITHUB_TOKEN && gh run download "$RUN_ID" --repo "$REPO" \
  -n "helm-values-*" \
  -D $LOG_DIR/helm-$RUN_ID/ 2>/dev/null

# Download full run logs
unset GITHUB_TOKEN && gh api "repos/$REPO/actions/runs/$RUN_ID/logs" \
  > $LOG_DIR/run-$RUN_ID.zip 2>/dev/null
cd $LOG_DIR && unzip -o "run-$RUN_ID.zip" -d "run-$RUN_ID" 2>/dev/null; cd -

2. Common Helmfile Failures

# Search for helmfile errors
grep -riE "error|FAIL|helm.*failed|release.*failed|timeout|timed out" \
  $LOG_DIR/run-$RUN_ID/ 2>/dev/null | grep -v "ignore-not-found" | head -30

Error Pattern	Cause	Fix
`release "X" failed`	Chart values incompatible	Check guide's `values*.yaml` against chart version
`timed out waiting for condition`	CRDs not installed	Ensure CRD install step ran before helmfile
`cannot re-use a name that is still in use`	Previous cleanup failed	Manual cleanup: `helm uninstall -n <ns>`
`Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists`	Orphaned resources	Delete the conflicting resource, then retry
`context deadline exceeded`	Helm timeout too short	Increase `--timeout` in helmfile args
`ImagePullBackOff` during deploy	GHCR auth or image tag missing	Check `image-metadata.json` artifact for resolved tags

3. Custom Script Failures (Kustomize+Helm)

For guides using custom_deploy_script (tiered-prefix-cache, wide-ep-lws):

# The custom script is inlined in the caller workflow — check the workflow file
grep -A 20 "custom_deploy_script:" $LOG_DIR/run-$RUN_ID/*.txt 2>/dev/null

Common issues:

kustomize build fails on invalid overlays
LeaderWorkerSet CRD not installed
Helm install needs --create-namespace but namespace already exists

4. WVA deploy/install.sh Failures

# Search for install.sh specific errors
grep -riE "install.sh|deploy/install" $LOG_DIR/run-$RUN_ID/ 2>/dev/null | tail -20

Common issues:

deploy/install.sh not found — caller repo checkout failed
Helm chart version mismatch between WVA and llm-d charts

Resolution

Transient: Rerun

If the failure looks transient (network timeout, mirror down):

unset GITHUB_TOKEN && gh run rerun "$RUN_ID" --repo "$REPO"

Chart/Values Change: Check Recent Commits

# What changed in llm-d/llm-d guides since last successful run?
LAST_SUCCESS_DATE=$(unset GITHUB_TOKEN && gh run list --repo llm-d/llm-d --limit 50 \
  --json workflowName,conclusion,startedAt \
  | jq -r '[.[] | select(.workflowName | test("<guide>")) | select(.conclusion == "success")][0].startedAt')

unset GITHUB_TOKEN && gh api "repos/llm-d/llm-d/commits?since=$LAST_SUCCESS_DATE&path=guides/<guide-name>" \
  | jq -r '.[] | "\(.sha[:8]) \(.commit.message | split("\n")[0])"'

File Issue

If the failure is a real regression:

unset GITHUB_TOKEN && gh issue create --repo llm-d/llm-d \
  --title "Nightly E2E: <guide> deploy failure on <platform>" \
  --body "Run: https://github.com/llm-d/llm-d/actions/runs/$RUN_ID
Failed step: <step-name>
Error: <error-summary>
Last successful: <date>"

Related Skills

nightly — Parent router
nightly:pods — If deploy succeeds but pods don't come up