isr-replication-drill

name: isr-replication-drill description: Run the ISR rolling-restart chaos drill for RobustMQ — 10 rounds of kill-one-node + write + restart + verify across a 3-replica shard. Use when the user wants to verify replica replication / ISR health, failover, or behaviour under churn — e.g. "演练 isr 副本同步", "验证三副本同步", "kill leader 演练", "验证 leader 切换/故障转移", "滚动重启/频繁切换 isr 是否正常", "test ISR replication", "副本同步是否正常".

RobustMQ ISR Replication Drill

Verifies ISR (In-Sync Replica) replication, leader failover, and recovery under repeated rolling restarts on a running 3-node cluster.

The single drill (three_replica_chaos_rolling_kill) covers all three flows the user cares about:

1. Write + observe — acks=all writes in full ISR; all 3 replicas leo==hw==cumulative, lso==0.
2. Kill (leader or follower), observe — ISR shrinks to 2; leader switches if victim was leader; epoch advances; survivors hold all committed data; acks=all still commits on 2 survivors.
3. Restart, observe follower role — victim rejoins ISR as follower (not leader); all 3 replicas catch up to cumulative; all nodes agree on leader/epoch/ISR/leo/hw.

Runs 10 rounds; victim rotates n1→n2→n3→n1…, covering both follower-kill and leader-kill scenarios in the same run.

When to Use

• Verify a 3-replica shard genuinely places replicas across all 3 nodes
• Confirm followers replicate the leader's log (LEO converges, HW advances)
• Verify leader failover: kill leader → switch + no committed-data loss + epoch bump
• Verify ISR recovery: killed node rejoins as follower, offsets fully caught up
• Regression-check fixed defects (see "Defects This Guards" below)

Key Semantics — what "ISR healthy" means

Storage type. ISR replication only works with EngineRocksDB or EngineMemory. EngineSegment returns NotLeaderForPartition on follower fetch — use EngineRocksDB.

HW lags by one fetch round. A follower learns the leader's HW from the next fetch response. Always poll until convergence — never assert HW immediately after a write.

Prerequisites

# Build the executable (NOT `cargo build -p broker-server` — that only builds the lib).
cargo build --bin broker-server

Cluster configs: config/cluster/server-{1,2,3}.toml (broker_id 1/2/3; meta/grpc 1228/2228/3228; engine tcp 1779/2779/3779; admin http 58080/58082/58083). Data dirs: data/broker-{1,2,3}.

Test: tests/tests/engine/isr.rs::three_replica_chaos_rolling_kill (#[ignore]). Test package: robustmq-test.

Drill Steps

0. Clean state (mandatory)

for c in 1 2 3; do p=$(pgrep -f "server-$c.toml"); [ -n "$p" ] && kill -INT "$p"; done
for c in 1 2 3; do while pgrep -f "server-$c.toml" >/dev/null; do sleep 1; done; done
rm -rf data/broker-1 data/broker-2 data/broker-3 data/logs /tmp/n1.log /tmp/n2.log /tmp/n3.log

NEVER pkill -9 — SIGKILL leaves an ESTABLISHED socket on port 1228 that mac_agent keeps alive, blocking the next bind. Always stop with kill -INT.

1. Start the 3-node cluster (staggered so node1 bootstraps first)

./target/debug/broker-server --conf config/cluster/server-1.toml > /tmp/n1.log 2>&1 &
sleep 8
./target/debug/broker-server --conf config/cluster/server-2.toml > /tmp/n2.log 2>&1 &
sleep 9
./target/debug/broker-server --conf config/cluster/server-3.toml > /tmp/n3.log 2>&1 &
sleep 13

Verify:

grep -oE "bootstrapping single-node cluster|Successfully joined cluster via peer|Meta Service cluster is ready|Failed to start GRPC" /tmp/n1.log /tmp/n2.log /tmp/n3.log | sort | uniq -c
curl -s http://localhost:58080/api/info | python3 -c "import sys,json;d=json.load(sys.stdin).get('data',{});print('nodes:',sorted(n.get('node_id') for n in d.get('broker_node_list',[])))"

Expect: node1 bootstrap, node2/3 joined, nodes: [1, 2, 3].

2. Run the drill

cargo test -p robustmq-test three_replica_chaos_rolling_kill -- --ignored --nocapture

Each round:

1. wait_full_and_caught_up — precondition: all 3 replicas leo==hw==cumulative, lso==0, in ISR.
2. write_acks_all(BATCH=30) → cumulative += 30. Confirm all 3 replicas caught up.
3. kill_node(victim) (victim rotates n1→n2→n3→…).
4. Wait for victim to leave ISR; if was-leader, assert new leader elected, epoch advanced.
5. Assert survivors: available, leo==hw==cumulative, lso==0, replicas=={1,2,3}.
6. write_acks_all(BATCH=30) on 2 survivors (degraded ISR). Confirm both commit.
7. restart_node(victim).
8. wait_full_and_caught_up — all 3 replicas back in ISR, fully caught up.
9. Assert restarted node is follower (not the new leader).
10. Cross-node check: all 3 nodes agree on leader/epoch/ISR/leo/hw.
11. Log blacklist scan.

After all 10 rounds: read back all cumulative records; final log scan.

Expected tail:

===== round 1: leader=n3, restarting n1 (was_leader=false), cumulative=30 =====
  n1 left ISR after 11.3s: leader=n3 epoch=0 isr=[2, 3]
  degraded write committed on 2 survivors at leo=hw=60
===== round 1 OK: leader=n3 epoch=0 isr=[1, 2, 3] all leo=hw=60 lso=0 =====
...
===== round 3: leader=n3, restarting n3 (was_leader=true), cumulative=150 =====
  n3 left ISR after 28.3s: leader=n1 epoch=1 isr=[1]
  degraded write committed on 2 survivors at leo=hw=180
===== round 3 OK: leader=n1 epoch=1 isr=[1, 2, 3] all leo=hw=180 lso=0 =====
...
CHAOS COMPLETE: 10 rounds, 600 committed records read back OK, 6 leader-epoch switches, all replicas consistent

Timings:

• Dead follower leaves ISR in ~6–11s (replica_lag_time_max_ms).
• Dead leader triggers switch in ~28–30s (heartbeat timeout).
• Follower rejoin after restart in ~4–10s (reconcile self-heal).

Check Conditions (per round)

Log Blacklist

Lines that must NEVER appear (each is an unambiguous defect):

Benign noise (NOT blacklisted): Unreachable node, fetcher retry, reconcile: follower has no fetcher, diverged-tail truncation on leader switch.

Quick scan:

for c in 1 2 3; do echo "=== n$c ==="; grep -E "invalid state: expect|last_log_id=None|Clean the hole|forcing exit|acks=all timed out|NotEnoughReplicas|panicked at" /tmp/n$c.log | head -5; done

Pass / Fail Criteria

PASS only if all hold after all 10 rounds:

• All 10 rounds complete without assertion failure.
• Every replica: available, in_isr, leo == hw == cumulative, lso == 0.
• segment.replicas always [1, 2, 3].
• leader_epoch monotonic across all rounds; advances on leader-kill rounds.
• All 3 nodes agree on leader/epoch/ISR/leo/hw after every round.
• Final read-back returns exactly cumulative (= ROUNDS × 2 × BATCH = 600) records.
• No blacklisted lines in any node log.

Common FAIL signatures:

Defects This Drill Guards

Each defect below was found by this drill; a regression re-introduces it and the drill fails.

Cleanup

for c in 1 2 3; do p=$(pgrep -f "server-$c.toml"); [ -n "$p" ] && kill -INT "$p"; done
for c in 1 2 3; do while pgrep -f "server-$c.toml" >/dev/null; do sleep 1; done; done

Troubleshooting

Run Report

After each drill session emit a structured summary:

=== ISR REPLICATION DRILL RUN REPORT ===
Date:        <YYYY-MM-DD>
Code commit: <git rev-parse --short HEAD>

Drill — rolling-restart chaos (10 rounds)
  Result:    PASS / FAIL
  Rounds:    <N>/10 passed
  Records:   <N> committed records read back OK
  Epochs:    <N> leader-epoch switches
  Details:   <any notable anomalies or the failure round + panic line>

Log errors:
  node1: <N> ERRORs  (grep -c ' ERROR' /tmp/n1.log)
  node2: <N> ERRORs
  node3: <N> ERRORs

Overall:  PASS / FAIL
========================================

Gather inputs:

git rev-parse --short HEAD
for c in 1 2 3; do echo "node$c ERROR: $(grep -c ' ERROR' /tmp/n$c.log)"; done