sagemaker-hyperpod

name: sagemaker-hyperpod description: | Amazon SageMaker HyperPod expert for ML training clusters with Trainium or GPU. Use when: creating HyperPod clusters, running distributed training, configuring EKS or Slurm orchestration, troubleshooting cluster issues, checking quotas, or when user mentions "hyperpod", "hyp", "ml-cluster", "trainium", "trn1", "distributed training", or "multi-node training". argument-hint: "[cluster-name or action]" context: fork model: sonnet skills:

• aws-mcp-setup allowed-tools:
• mcp__sagemaker__*
• mcp__aws-mcp__*
• mcp__awsdocs__*
• WebFetch
• Bash(hyp *)
• Bash(aws sagemaker *)
• Bash(kubectl *)
• Bash(aws eks *)
• Bash(aws ec2 describe-*)
• Bash(aws servicequotas *)
• Bash(aws s3 *)
• Bash(aws ssm start-session *)
• Bash(aws sts get-caller-identity)
• Bash(aws logs *)
• Bash(aws iam get-role*)
• Bash(aws iam list-*)
• Bash(helm *)
• Bash(pip install sagemaker-hyperpod) hooks: PreToolUse:
  • matcher: Bash(aws sagemaker create-cluster*) command: aws sts get-caller-identity --query Account --output text once: true
  • matcher: Bash(hyp create*) command: aws sts get-caller-identity --query Account --output text once: true

Amazon SageMaker HyperPod Expert

You are an expert in Amazon SageMaker HyperPod for provisioning resilient ML training clusters with AWS Trainium and NVIDIA GPUs.

When This Skill Activates

• Creating HyperPod clusters (EKS or Slurm)
• Running distributed ML training jobs
• Troubleshooting cluster issues
• Checking quotas or instance availability
• User mentions: "hyperpod", "hyp", "trainium", "trn1", "distributed training"

Detailed Guides

Orchestrator Selection

Instance Types

IMPORTANT: ml.trn1.2xlarge is NOT supported for HyperPod - only ml.trn1.32xlarge.

CRITICAL: Pre-Creation Validation

ALWAYS perform these checks BEFORE creating a cluster:

1. Verify Instance Type Support

# Must say "for cluster usage" in quota name
aws service-quotas list-service-quotas \
  --service-code sagemaker --region us-east-1 \
  --query 'Quotas[?contains(QuotaName, `<INSTANCE_TYPE>`) && contains(QuotaName, `cluster`)].[QuotaName,Value]' \
  --output table

2. Check AZ Availability

aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --filters Name=instance-type,Values=trn1.32xlarge \
  --region us-east-1 \
  --query 'InstanceTypeOfferings[*].Location' --output text

3. For EKS: Ensure 2+ AZs in config.yaml

availability_zone_ids:
  - use1-az6  # Primary for workers
  - use1-az4  # Secondary for EKS HA

4. Check K8s Version (EKS Only)

WebFetch: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar
Prompt: What is the latest Kubernetes version in standard support?

5. Check Add-on Compatibility (EKS Only)

Before upgrading K8s versions, verify HyperPod add-ons support the target version:

aws eks describe-addon-versions --addon-name amazon-sagemaker-hyperpod-taskgovernance \
  --query 'addons[0].addonVersions[*].compatibilities[*].clusterVersion' --output text

WARNING: EKS does NOT support downgrading. Stay on a supported version if you need HyperPod add-ons.

EKS Quick Start

# 1. Install CLI
pip install sagemaker-hyperpod

# 2. Initialize cluster stack
hyp init cluster-stack my-cluster
cd my-cluster

# 3. Edit config.yaml (ensure 2+ AZs!)

# 4. Validate and create
hyp validate && hyp create cluster-stack --region us-east-1

# 5. Set context
hyp set-cluster-context --cluster-name <NAME> --region us-east-1

Submit Training Job (EKS)

# Option 1: Using config file (recommended)
hyp init hyp-pytorch-job my-job
cd my-job
# Edit config.yaml
hyp validate
hyp create hyp-pytorch-job

# Option 2: Command line
hyp create hyp-pytorch-job \
  --job-name my-job \
  --image <ECR-IMAGE> \
  --instance-type ml.trn1.32xlarge \
  --node-count 1 \
  --accelerators 16 \
  --accelerators-limit 16

Monitor Training Job (EKS)

# List jobs
hyp list hyp-pytorch-job

# Job details
hyp describe hyp-pytorch-job --job-name <NAME>

# View logs
hyp get-logs hyp-pytorch-job --job-name <NAME> --follow

# List pods
hyp list-pods hyp-pytorch-job --job-name <NAME>

# Delete job
hyp delete hyp-pytorch-job --job-name <NAME>

Full guide: See orchestrators/eks/job-submission.md

Slurm Quick Start

# 1. Prepare lifecycle scripts (use AWS samples)
git clone https://github.com/aws-samples/awsome-distributed-training.git
cd awsome-distributed-training/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/

# 2. Upload to S3
aws s3 cp . s3://my-bucket/lifecycle-scripts/ --recursive

# 3. Create cluster
aws sagemaker create-cluster --cluster-name my-cluster \
  --instance-groups '[...]' --vpc-config "..."

# 4. Connect via SSM
aws ssm start-session --target <instance-id>

Full workflow: See reference/slurm-guide.md

Model Compatibility (Trainium/Inferentia)

CRITICAL: Verify model support before configuring Trainium jobs.

Check Support

WebFetch: https://huggingface.co/docs/optimum-neuron/en/supported_architectures
Prompt: List supported model architectures for training on Trainium

Currently Supported (Training)

Common Errors (Quick Reference)

Full troubleshooting: See reference/troubleshooting.md

Infrastructure Requirements

EFA Single-AZ Requirement

For EFA-enabled instances (trn1, p4d, p5), ALL instances MUST be in the SAME AZ.

Security Group

Must allow ALL traffic within itself:

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxx --protocol all --port -1 --source-group sg-xxx

CIDR Sizing

Quota Management

# Check quota
aws service-quotas get-service-quota \
  --service-code sagemaker --quota-code L-6865522E --region us-east-1

# Request increase
aws service-quotas request-service-quota-increase \
  --service-code sagemaker --quota-code L-6865522E --desired-value 4

Common codes:

• L-6865522E: ml.trn1.32xlarge for cluster usage
• L-5C4CD236: ml.p5.48xlarge for cluster usage

Diagnostic Commands

# Cluster status
aws sagemaker describe-cluster --cluster-name NAME

# List nodes
aws sagemaker list-cluster-nodes --cluster-name NAME

# CloudWatch logs
aws logs get-log-events \
  --log-group-name /aws/sagemaker/Clusters/NAME/ID \
  --log-stream-name LifecycleConfig/GROUP/INSTANCE

# EKS nodes/pods
kubectl get nodes && kubectl get pods -A