hpc

The parallel partition requires MinNodes=2 — it will reject single-node jobs with "Node count specification invalid". It is designed for MPI jobs that span multiple nodes.

Wrong: #SBATCH --partition=parallel for array jobs → submission fails Right: #SBATCH --partition=standard for array jobs → 301 nodes available

When to Use Each Partition

standard (default choice for most research computing):

• Embarrassingly parallel work: array jobs where each task is independent (PIN estimation, file parsing, per-stock/per-year processing)
• Single-node Python/R with ProcessPoolExecutor, multiprocessing, mclapply
• Any job where tasks don't communicate with each other
• MaxNodes=1, so each array element runs on exactly one node

parallel (multi-node distributed computing):

• MPI jobs where processes on different nodes exchange messages (mpi4py, OpenMPI, MVAPICH)
• Dask distributed or Ray clusters spanning multiple nodes
• Large linear algebra / matrix factorizations that exceed single-node RAM (ScaLAPACK, PETSc)
• Simulations with inter-process communication (CFD, molecular dynamics)
• Key requirement: your code must explicitly coordinate across nodes (MPI, Dask scheduler, etc.) — ProcessPoolExecutor and multiprocessing are single-node only
• MinNodes=2, 96 CPUs and 768GB RAM per node — use when one node isn't enough

gpu (GPU-accelerated workloads):

• Deep learning training/inference (PyTorch, TensorFlow, JAX)
• GPU-accelerated linear algebra (CuPy, RAPIDS)
• LLM inference or fine-tuning

interactive (debugging and development):

• Testing job scripts before full submission: salloc -p interactive --cpus-per-task=4 --mem=16G --time=1:00:00
• Debugging segfaults or data loading issues
• 12-hour max — not for production runs

Python/R Environment

• pixi: Install to $HOME/.pixi/bin/pixi via curl -fsSL https://pixi.sh/install.sh | bash
• Project envs: $HOME/projects/<name>/.pixi/envs/default/bin/python
• Modules (alternative): module load python — but pixi preferred for reproducibility
• NEVER install Jupyter kernels globally on HPC

Slurm Job Submission

Basic Submit Script

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=3:00:00
#SBATCH --output=logs/job-%A_%a.log

mkdir -p logs

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

PYTHON=$HOME/projects/my-project/.pixi/envs/default/bin/python
$PYTHON -u my_script.py --workers ${SLURM_CPUS_PER_TASK:-8}

Submission

sbatch script.sh              # submit
sbatch script.sh arg1 arg2    # args passed to script as $1, $2

Note: unlike SGE's qsub run.sh <model>, Slurm passes arguments after the script name directly. Use ${1:?Usage: sbatch script.sh <arg>} to enforce.

Array Jobs

Pattern

#SBATCH --array=1-176           # tasks 1 through 176
#SBATCH --array=1-176%50        # max 50 concurrent tasks
#SBATCH --array=1,5,9,13        # specific tasks only

Year × Chunk Sharding (PIN estimation pattern)

#SBATCH --array=1-176
# 22 years × 8 chunks = 176 tasks
# Decode: year = START_YEAR + (id-1)/NCHUNKS, chunk = (id-1)%NCHUNKS

NCHUNKS=8
START_YEAR=2003

idx=$((SLURM_ARRAY_TASK_ID - 1))
year=$((START_YEAR + idx / NCHUNKS))
chunk=$((idx % NCHUNKS))

Task List Sharding (file processing pattern)

# Equivalent to SGE's sed -n "${SGE_TASK_ID}p" pattern
ITEM=$(sed -n "${SLURM_ARRAY_TASK_ID}p" "$TASK_LIST")

Re-running Failed Tasks

# Re-run specific tasks
sbatch --array=5,12,87 script.sh

# Re-run a range
sbatch --array=10-20 script.sh

SGE to Slurm Translation

Directives

Environment Variables

Commands

Conversion Checklist

When converting an SGE script to Slurm:

1. Replace #$ directives with #SBATCH equivalents (see table above)
2. Add #SBATCH --partition=standard (SGE has no equivalent — partition is implicit)
3. Add #SBATCH --time= (SGE defaults to unlimited on WRDS)
4. Replace $SGE_TASK_ID → $SLURM_ARRAY_TASK_ID
5. Replace $NSLOTS → $SLURM_CPUS_PER_TASK
6. Replace $JOB_ID → $SLURM_JOB_ID
7. Remove #$ -cwd and #$ -j y (Slurm defaults)
8. Update log path variables: $TASK_ID → %a, $JOB_ID → %A
9. Update data paths from WRDS scratch to HPC scratch

Monitoring & Debugging

Check Job Status

squeue -u $USER                              # all my jobs
squeue -j 12345678                           # specific job
squeue -j 12345678 -t R | wc -l             # count running tasks
squeue -j 12345678 -t PD                     # show pending tasks + reasons
squeue -u $USER --format='%.10i %.9P %.12j %.2t %.10M %.4C %R'  # detailed

Common Pending Reasons

Job Accounting (after completion)

sacct -j 12345678 --format=JobID,State,ExitCode,Elapsed,MaxRSS,NCPUS
sacct -j 12345678 -a --format=JobID,State,ExitCode  # all array tasks

Log Files

Output goes to --output path. With %A_%a pattern:

• logs/est-12345678_1.log — job 12345678, array task 1
• Check for errors: grep -rl 'Error\|Traceback' logs/est-12345678_*.log

Resource Billing

UVA HPC bills in Service Units (SUs), which are weighted CPU-core-hours:

SU = (CPU_cores × 4.6369 + Memory_GB × 0.2842) × hours

Cost Examples (standard partition)

With 10M SUs allocated, even aggressive usage (8 CPU × 176 tasks × 3 hrs = ~24K SUs) is negligible (<0.25% of allocation).

Check Balance

allocations                    # show allocation balance
allocations -a myallocation    # specific allocation

WRDS Data Access

WRDS PostgreSQL is accessible from HPC compute nodes. Use polars + connectorx for fast data pipelines that replace SAS entirely.

Connection

• Host: wrds-pgdata.wharton.upenn.edu:9737
• Credentials: ~/.pgpass (chmod 600)
• User: edwin_hu (UVA account)

Quick Start

from wrds_conn import read_wrds

# WRDS SQL → polars DataFrame in one line
df = read_wrds("SELECT * FROM crsp.msf WHERE date >= '2020-01-01'")

# Write to Parquet for reuse
df.write_parquet("/scratch/vwh7mb/data/crsp_msf.parquet")

wrds_conn.py (see examples/wrds_conn.py) parses .pgpass and builds a connectorx-compatible URI — connectorx doesn't read .pgpass natively.

Pipeline: SQL → polars → Parquet (replaces SAS)

Old: WRDS SAS → .sas7bdat (7GB) → Python HDF5 conversion → .h5 (390MB)
New: WRDS PostgreSQL → polars/connectorx → .parquet

No SAS license needed. Single step. Portable output.

See references/wrds-polars-pipeline.md for full examples (joins, partitioned output, Slurm submission for large queries).