run-roachtest

name: run-roachtest description: Run a single CockroachDB roachtest end-to-end: pick local vs. user's GCE worker, launch detached on worker via tmux + roachstress.sh with long-poll done-notification and tail. Use whenever user asks to run/stress/kick off a roachtest, or just modified one and next step is running it. Single test + single iteration only; nightly loops belong elsewhere.

run-roachtest

Run a CockroachDB roachtest from end to end. The skill handles three things that are easy to get wrong:

1. Where to run (local vs. the user's GCE worker).
2. How to launch on the worker so the run survives a closed laptop lid and Claude stays informed when it finishes.
3. How to give the user a no-friction way to peek at progress.

Requires the roachdev worker CLI configured (provisions and addresses the user's GCE worker without a name argument).

When to use this skill

• The user says "run X", "kick off X", "stress X", "try X on the worker", "run this test", etc., where X is a roachtest.
• The user just wrote or modified a roachtest and the next move is to run it.

Do NOT use this skill for:

• Multi-test sweeps or nightly-style loops (different shape — wrap many invocations).
• Builds/unit tests (use ./dev test).
• Investigating an existing failed roachtest run (use the teamcity / engflow-artifacts skills).

Step 1 — Decide local vs. remote

Pick local or remote as follows.

The user's wording is the strongest signal. Honor it directly:

When wording is silent, read the test source under pkg/cmd/roachtest/tests/ and consider:

• Linux-only mechanics — cgroups, iptables, disk stalls, dmsetup, network namespaces, sysctls. → remote.
• Heavy clusters — anything provisioning more than a couple of nodes, or specifying high spec.CPU(...). Local mac will choke. → remote.
• Local-mode bypass — many heavy tests check c.IsLocal() and short-circuit to a smoke test. If present, local IS viable; warn the user it will be a smoke test rather than the real thing.
• Toy/acceptance tests — acceptance/* and similarly named tests are usually fine locally.

If after looking at the test you still cannot tell, ask the user. A 30-second clarification beats a 2-hour run on the wrong machine.

Step 2a — Local run

Use roachstress.sh -l from the repo root. It builds locally and runs in-process:

bash pkg/cmd/roachtest/roachstress.sh -c 1 -u -l '<test-name>'

Tee to a log so output is inspectable mid-run. Run as a background Bash task (run_in_background: true). No tmux needed — local runs inherit the user's session.

That's it for local. The rest of the skill is the remote workflow.

Step 2b — Remote run on the user's GCE worker

roachdev worker ssh (and the other roachdev worker subcommands) default to the current user's worker when no name is given — e.g. roachdev worker ssh -- 'hostname' returns the user's worker hostname. Always omit the worker name so this skill works unmodified for every developer.

The repo on the worker lives at ~/go/src/github.com/cockroachdb/cockroach (standard $GOPATH layout that roachdev provisions).

Step 2b.1 — Pre-flight

Run these in parallel via SSH (single round-trip):

roachdev worker ssh -- '
  tmux ls 2>&1;
  echo ---;
  pgrep -af roachstress | head -10;
  echo ---;
  cd ~/go/src/github.com/cockroachdb/cockroach && git rev-parse --abbrev-ref HEAD && git rev-parse HEAD
'

Inspect the output:

• Worker stopped — roachdev worker resume first.
• Existing tmux sessions — note their names so the new session name does not collide.
• Running roachstress — another agent or the user is in the middle of a test. Do NOT wipe artifacts/ or destroy the worker. Pick a unique session/log/signal name. Before touching the checkout (branch switch, fetch, reset), confirm those tests are past their build phase — see Step 2b.1.5.
• Branch / SHA — see "Getting your code on the worker" below.

Step 2b.1.5 — Is the in-progress test past its build phase?

If pre-flight showed another roachstress already running, before touching the checkout (branch switch, fetch, reset) you must confirm its build phase is done — otherwise you corrupt its build.

See references/concurrent-runs.md for the check command, the "build is done" criteria, and the long-poll wait-for-build loop. Read it only when this case applies; the common case (no concurrent run) skips straight to Step 2b.2.

Step 2b.2 — Getting your code on the worker

If the test is unmodified upstream code (e.g. acceptance/build-info), just use whatever branch/SHA the worker happens to be on. roachstress.sh caches binaries by SHA, so reusing the existing checkout reuses cached binaries — fast startup.

If the test depends on local changes:

1. Push the branch from the local worktree to the user's GitHub fork: git push -u origin <branch>.
2. On the worker, fetch and check out:

   roachdev worker ssh -- '
     cd ~/go/src/github.com/cockroachdb/cockroach &&
     git fetch origin <branch> &&
     git checkout <branch> &&
     git reset --hard origin/<branch>
   '
   

   This is safe even with other tests running, provided Step 2b.1.5 confirmed they are past the build phase. The new SHA gets a fresh artifacts/<new-sha>/ directory.

Worktree fallback (rare). If you cannot wait for the in-progress build to finish, git worktree add ../cockroach-<branch> origin/<branch> and run roachstress.sh from the worktree path. Downside: bazel cache contention slows both builds, and the worktree needs cleanup later.

Step 2b.3 — Launch in detached tmux

Pick three unique names so concurrent runs do not collide:

For example: rt-build-info, ~/rt-build-info.log, rt-build-info-done.

Launch:

roachdev worker ssh -- "tmux new-session -d -s <SESSION> \"cd <REPO> && bash pkg/cmd/roachtest/roachstress.sh -c 1 -u <TEST> 2>&1 | tee <LOG>; echo EXIT=\\\$? >> <LOG>; tmux wait-for -S <SIGNAL>\""

Why this exact form:

• tmux new-session -d detaches immediately — SSH returns, the run keeps going. Closing the laptop is fine.
• bash pkg/cmd/roachtest/roachstress.sh from the repo root: the script demands its own working directory.
• -u: build the full cockroach binary with the DB Console UI baked in (instead of cockroach-short). This makes the cluster's admin UI available — useful when you want to hand the user a clickable http://...:26258 link for live debugging. Adds ~1-2 min to the first build per SHA; cached afterward.
• 2>&1 | tee <LOG>: captures stdout + stderr into a single inspectable file.
• echo EXIT=$? >> <LOG>: records the script's exit code so the notification can report success/failure without re-grepping.
• tmux wait-for -S <SIGNAL>: signals a tmux semaphore as the very last thing. Step 2b.4 blocks on the matching wait-for <SIGNAL>.
• The \\\$? escaping: outer SSH consumes one layer of \, the outer shell another — three backslashes leave one literal \$? for tmux to pass through to bash so the exit-code expansion happens inside tmux's window, not at SSH-time.

After launch, verify the session is alive:

roachdev worker ssh -- 'tmux ls | grep <SESSION>; sleep 5; tail -30 <LOG>'

The early log should show roachstress.sh locating binaries and starting the test.

Step 2b.4 — Long-poll notification

Spawn a Bash task in background that blocks on the tmux semaphore. When the run finishes, the SSH command returns, the background task completes, and Claude gets a task-completion notification.

roachdev worker ssh -- 'tmux wait-for <SIGNAL> && echo SIGNAL_RECEIVED && tail -40 <LOG>'

Run with run_in_background: true. Set the Bash timeout to the expected max duration plus a safety margin (e.g. 6h for a long perturbation test, 10min for acceptance/build-info).

This is long-poll, not poll: the worker-side tmux wait-for blocks indefinitely on a kernel-cheap semaphore wait. There is no periodic checking — Claude simply gets one notification at the end.

When the notification arrives:

• Read the task output (it ends with the last 40 log lines + EXIT code).
• Summarize for the user: pass/fail, duration, and where the artifacts live (artifacts/<sha>/<runid>/<test-name>/run_1/).

Step 2b.5 — Long-running tail (peek window)

Spawn a second background Bash task that runs tail -F against the log:

roachdev worker ssh -- 'tail -F <LOG>'

Run with run_in_background: true and a long timeout. This task is NOT meant to complete; it lets the user (and Claude) read live output through BashOutput / /tasks without re-SSHing each time.

Tell the user the task ID so they know which background task to peek at.

Step 3 — Reporting back

Initial message after launch — keep it short:

Launched <test> on the worker.
- tmux session: <SESSION>
- log: <LOG>
- notify task: <ID>      (fires when run finishes)
- tail task:   <ID>      (live log, peek anytime)

When the notify task fires, post a one-paragraph result: pass/fail, duration, key error if it failed, artifact path on the worker, and a suggested roachdev worker scp command to download anything specific the user might want.

Pitfalls and gotchas

• Closing SSH does not stop tmux. tmux new-session -d detaches before SSH returns. The run is robust to lid-close, network drop, and Claude's bash timeout.
• Concurrent runs share artifacts/. Each roachstress.sh invocation gets its own artifacts/<sha>/<runid>/ subdir, so collision is fine. But rm -rf artifacts/ would nuke another agent's work — never do this without checking pgrep -af roachstress.
• Branch switches affect anyone building from the same checkout. If another test is still in its build phase, switching breaks it. See Step 2b.1.5 — wait for the build to finish, then switch in place. Worktrees are a last resort.
• ./dev vs ./pkg/cmd/roachtest/roachstress.sh. Always use roachstress.sh for roachtests — it cross-compiles cockroach, workload, and roachtest for the cluster's arch and caches by SHA. ./dev test is for unit tests and will not produce the right binaries.

Artifact paths

After a successful run, artifacts live at:

~/go/src/github.com/cockroachdb/cockroach/artifacts/<sha>/<runid>/<test-name>/run_1/

Useful files inside run_1/:

• test.log — main test log; start here when debugging.
• <N>.perf/stats.json — workload per-second tick data (node <N> is the workload node, usually the highest-numbered).
• <N>.perf/phases.json — perturbation baseline / perturbation / recovery boundaries (RFC3339Nano).
• 1.perf/stats.json — aggregated ratio summary stats (perturbation tests only).
• run_*.log — individual command logs (cockroach start, workload invocations, etc).

Download with:

roachdev worker scp -- ':<path-on-worker>' /local/path

Quick reference: full remote launch

# 1. Pre-flight.
roachdev worker ssh -- '
  tmux ls; echo ---;
  pgrep -af roachstress;
  echo ---;
  cd ~/go/src/github.com/cockroachdb/cockroach && git rev-parse HEAD
'

# 2. (If needed) push and fetch your branch.
git push -u origin <branch>
roachdev worker ssh -- '
  cd ~/go/src/github.com/cockroachdb/cockroach &&
  git fetch origin <branch> &&
  git reset --hard origin/<branch>
'

# 3. Launch detached.
roachdev worker ssh -- "tmux new-session -d -s rt-<name> \"cd ~/go/src/github.com/cockroachdb/cockroach && bash pkg/cmd/roachtest/roachstress.sh -c 1 -u <test> 2>&1 | tee ~/rt-<name>.log; echo EXIT=\\\$? >> ~/rt-<name>.log; tmux wait-for -S rt-<name>-done\""

# 4. Notify task (run_in_background: true).
roachdev worker ssh -- 'tmux wait-for rt-<name>-done && echo SIGNAL_RECEIVED && tail -40 ~/rt-<name>.log'

# 5. Tail task (run_in_background: true).
roachdev worker ssh -- 'tail -F ~/rt-<name>.log'