devnet-debug

name: devnet-debug description: Debug a Kurtosis devnet that is not producing blocks. Iterates through logs → code correlation → fix → rebuild → retest until the chain advances.

Devnet Debug Skill

Debug a stalled or misbehaving eth2030 Kurtosis devnet. Covers two distinct failure modes: chain not advancing (Engine API bugs) and transactions not confirming (RPC format bugs). Both require different debugging paths.

Prerequisites

kurtosis engine start
docker build -t eth2030:local pkg/

Step 0 — Boot and observe

cd pkg/devnet/kurtosis
./scripts/cleanup.sh eth2030-devnet
./scripts/run-devnet.sh full-feature   # or single-client for faster iteration

# Check if the devnet had boot
kurtosis enclave inspect eth2030-devnet # if return `No enclave found with identifier 'eth2030-devnet'`, means no devnet

Step 1 — Classify the failure

Got the basic info for enclave:

kurtosis enclave inspect eth2030-devnet

It will show status, Files Artifacts and User Services, Note key services for eth devnet:

• cl-{N}-{cltpye}-geth: cl nodes, cltpye will depend on cl_type in config for kurtosis devnet
• el-{N}-geth-{cltpye}: el nodes, which is run by eth2030
• dora: a explorer for cl
• spamoor: send test tx

Run these three checks to pick the right debug path:

# 1. Block number advancing?
cast bn -r http://$(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc)

# 2. Spamoor confirming txs? (look for "0 tx confirmed" or error lines)
kurtosis service logs eth2030-devnet spamoor 2>&1 | grep -E "error|0 tx confirmed|receipts" | tail -20

# 3. CL verified on every slot?
kurtosis service logs eth2030-devnet cl-1-lighthouse-geth 2>&1 | grep -E "verified|error|ERROR" | tail -20

if some log not show can use -n 3000 to show 3000 line, or use -f to follow:

 kurtosis service logs eth2030-devnet el-2-geth-lighthouse -h
Show logs for a service inside an enclave

Usage:
  kurtosis service logs [flags] enclave [service...]

Flags:
  -a, --all                  Gets all logs.
  -x, --all-services         Returns service log streams for all logs in an enclave
  -f, --follow               Continues to follow the logs until stopped
  -h, --help                 help for logs
  -v, --invert-match         Inverts the filter condition specified by either 'match' or 'regex-match'. Log lines NOT containing match/regex-match will be returned
      --match string         Filter the log lines returning only those containing this match. Important: match and regex-match flags cannot be used at the same time. You should either use one or the other.
  -n, --num uint32           Get the last X log lines. (default 200)
      --regex-match string   Filter the log lines returning only those containing this regex expression match (re2 syntax regex may be used, more here: https://github.com/google/re2/wiki/Syntax). This filter will always work but will have degraded performance for tokens. Important: match and regex-match flags cannot be used at the same time. You should either use one or the other.

for ports, we can see the services info from kurtosis enclave inspect eth2030-devnet:

5de758969582   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:32771       RUNNING
                                                                metrics: 9001/tcp -> http://127.0.0.1:32772   
                                                                rpc: 8545/tcp -> 127.0.0.1:32769              
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:32773   
                                                                udp-discovery: 30303/udp                      
                                                                ws: 8546/tcp -> 127.0.0.1:32770  

we can use this cmd to got rpc endpoint:

kurtosis port print eth2030-devnet el-1-geth-lighthouse metrics
http://127.0.0.1:32772

for example, u can use cast -r;

cast block -r $(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc)  finalized

Path A — Engine API: chain not advancing

A1. Collect Engine API errors

# CL: what error code did Lighthouse receive?
kurtosis service logs eth2030-devnet cl-1-lighthouse-geth 2>&1 \
  | grep -E "error=|ERROR|invalid|WARN" | tail -40

# EL: what did eth2030 log for engine_ calls?
kurtosis service logs eth2030-devnet el-1-geth-lighthouse 2>&1 \
  | grep -E "engine_|WARN|ERROR" | tail -40

CL error code quick-reference:

A2. Engine API lifecycle checklist

Each slot: FCU(attrs) → build → getPayload → newPayload → FCU(no attrs)

getPayload JSON null vs [] checklist — Go nil slice → JSON null → CL rejects:

// Every array field must be non-nil even when empty:
Transactions  = make([][]byte, 0, len(txs))  // never nil
Withdrawals   = []*Withdrawal{}               // nil-guard before loop
BlobsBundle.Commitments/Proofs/Blobs = make([]hexBytes, len(in))
ExecutionRequests = [][]byte{}

Withdrawals nil after decode (Shanghai+ required):

// WRONG: loop over nil gives nil result; fails block validator
var withdrawals []*types.Withdrawal
for _, w := range payload.Withdrawals { ... }

// CORRECT: preserve non-nil for empty arrays
withdrawals = make([]*types.Withdrawal, 0, len(payload.Withdrawals))
for _, w := range payload.Withdrawals { ... }

Block hash mismatch — field checklist:

Every optional header field shifts the RLP hash. When reconstructing from a payload, always populate all fields the builder set:

GetPayload fork gate bugs — removing fork checks often fixes phantom rejections:

// WRONG: IsPrague check on an Amsterdam payload
func (s *Server) GetPayloadV4(...) {
    if !s.chain.Config().IsPrague(blockTimestamp) { return nil, ErrUnsupportedFork }
}
// CORRECT: let the payload ID resolve to the right version; no extra fork gate needed

Payload consumed on first read — always fatal for fork-choice:

// WRONG: CL calls getPayload twice (once to publish, once for newPayload)
p := s.payloads[id]
delete(s.payloads, id)  // BUG: second call returns nil → -32001

// CORRECT: remove the delete; expire via TTL or FCU

Path B — RPC: chain advancing but spamoor fails

B1. Understand spamoor's decode flow

Spamoor's "block X has 0 receipts, expected N" is almost never a receipt storage problem. The real cause is almost always transaction JSON decode failure upstream.

Spamoor flow (in spamoor/txpool.go):

1. eth_getBlockByNumber(..., true) → raw JSON block
2. For each tx: json.Unmarshal(rawTx, &types.Transaction) — failure → tx → txSkipMap
3. eth_getBlockReceipts → filter out receipts at txSkipMap indices
4. len(filtered) != txCount → "block X has 0 receipts, expected N"

How to confirm: call RPC directly and inspect tx fields:

curl -s -X POST http://$(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc) \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest",true],"id":1}' \
  | jq '.result.transactions[0] | keys'
# Must include: chainId, maxFeePerGas, maxPriorityFeePerGas for type-2 txs
# Missing any of these = "missing required field 'chainId' in transaction"

B2. RPCTransaction required fields

go-ethereum Transaction.UnmarshalJSON has strict required field checks per tx type. Missing any required field silently fails in spamoor's decode.

Complete RPCTransaction struct must include:

type RPCTransaction struct {
    // always present
    Hash, Nonce, From, Value, Gas, GasPrice, Input, Type string
    V, R, S  string  // must use tx.RawSignatureValues(), NOT hardcoded "0x0"
    BlockHash, BlockNumber, TransactionIndex *string
    To *string
    // EIP-2930+ (types 1,2,3,4)
    ChainID    *string          `json:"chainId,omitempty"`
    AccessList []RPCAccessTuple `json:"accessList,omitempty"`
    // EIP-1559+ (types 2,3,4)
    MaxFeePerGas         *string `json:"maxFeePerGas,omitempty"`
    MaxPriorityFeePerGas *string `json:"maxPriorityFeePerGas,omitempty"`
    // EIP-4844 (type 3)
    MaxFeePerBlobGas    *string  `json:"maxFeePerBlobGas,omitempty"`
    BlobVersionedHashes []string `json:"blobVersionedHashes,omitempty"`
    // EIP-7702 (type 4)
    AuthorizationList []RPCAuthorization `json:"authorizationList,omitempty"`
}

B3. Other RPC format bugs checklist

Block responses — missing fields break ethclient block parsing:

• Always: sha3Uncles, nonce, mixHash, totalDifficulty, size
• EIP-1559+: baseFeePerGas
• Shanghai+: withdrawals (emit [] not omitted), withdrawalsRoot
• Cancun+: blobGasUsed, excessBlobGas
• Cancun+: parentBeaconBlockRoot
• Prague+: requestsHash

eth_getBlockReceipts — must handle all parameter forms:

"latest" / "0x1a"             → string block tag / number
"0x<66 hex chars>"            → block hash (66-char string)
{"blockHash": "0x..."}        → object form (go-ethereum ethclient default)
{"blockNumber": "0x..."}      → object form with number

eth_sendRawTransaction — must RLP-decode, not JSON-unmarshal:

// WRONG: json.Unmarshal on a hex string
// CORRECT: bytes, _ := hex.DecodeString(rawHex[2:]); tx.DecodeRLP(bytes)

Rate limiting — 0 must mean unlimited:

if limit > 0 && count > limit { return rateLimitedError }

JSON-RPC 2.0 null result — result field must always be present:

// WRONG: nil result + omitempty → "result" field absent → spec violation
// CORRECT:
func successResponse(id json.RawMessage, result interface{}) *Response {
    if result == nil { result = json.RawMessage("null") }
    ...
}

B4. Container networking bugs

Bind addresses — all servers must bind 0.0.0.0 in Docker, not 127.0.0.1. Default 127.0.0.1 silently works locally but breaks container-to-container calls (ethereum-package, spamoor, assertoor, dora all connect from separate containers).

admin_nodeInfo must be reachable on port 8545 — ethereum-package polls this for up to 30 minutes waiting for a non-empty enode URL. If admin_ is only wired on a separate admin port, devnet startup stalls silently.

# Verify admin_ is routed on the standard RPC port
curl -s -X POST http://$(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc) \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"admin_nodeInfo","params":[],"id":1}' \
  | jq '.result.enode'
# Must be non-empty "enode://..." string

Path C — Consensus / finality

These are normal in short devnet runs; only investigate if persisting after 5+ min:

Core state bugs (harder to spot)

These manifest as wrong behavior after the chain is running:

tx.Sender() always nil — sender cache is never populated unless you explicitly recover the ECDSA signature after block insertion. Symptom: eth_getTransactionByHash returns from: "0x0000...0000".

Typed-nil interface panic — assigning a typed nil pointer to an interface gives a non-nil interface value, but calling a method on it panics:

var tracker *BALTracker = nil
var iface BALTrackerInterface = tracker  // iface != nil, but methods panic!
// CORRECT: if tracker != nil { iface = tracker }

Txpool nonces not advancing — after inserting a block, call txpool.SetHead(block) so the pool evicts confirmed txs and advances pending nonces. Without this, spamoor's second tx gets "nonce too low".

Receipt / txlookup index miss — in-memory index may miss receipts after restart. Always implement rawdb fallback:

func (b *Backend) GetReceipts(hash Hash) []*Receipt {
    if r := b.receiptCache[hash]; r != nil { return r }
    return rawdb.ReadReceipts(b.db, hash)
}

Targeted logging (add temporarily, remove before merge)

// Engine API: hash mismatch
slog.Warn("newPayload: hash mismatch",
    "computed", block.Hash(), "payload", payload.BlockHash,
    "parentBeaconRoot", parentBeaconRoot, "requestsHash", requestsHash)

// RPC: receipt lookup tracing
slog.Info("getBlockReceipts",
    "raw", string(raw), "blockHash", blockHash,
    "headerFound", header != nil, "receiptCount", len(receipts))

Full debug cycle

# 1. Start devnet
cd pkg/devnet/kurtosis
./scripts/cleanup.sh eth2030-devnet && ./scripts/run-devnet.sh full-feature

# 2. Check primary signals (30s after start)
sleep 30
cast bn -r http://$(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc)
kurtosis service logs eth2030-devnet spamoor 2>&1 | grep -E "error|confirmed" | tail -10
kurtosis service logs eth2030-devnet cl-1-lighthouse-geth 2>&1 | grep "verified\|ERROR" | tail -10

# 3. Path A: block = 0
kurtosis service logs eth2030-devnet el-1-geth-lighthouse 2>&1 | grep -E "engine_|WARN|ERROR" | tail -40

# 4. Path B: block > 0 but spamoor errors
kurtosis service logs eth2030-devnet spamoor 2>&1 | grep -E "error|receipt" | tail -20
curl -s -X POST http://$(kurtosis port print eth2030-devnet el-1-geth-lighthouse rpc) \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest",true],"id":1}' \
  | jq '.result.transactions[0] | keys'

# 5. Fix → compile → unit test → commit
cd pkg && go build ./... && go test ./rpc/... ./engine/... ./node/...

# 6. Rebuild + retest
docker build -t eth2030:local . -q
./scripts/cleanup.sh eth2030-devnet && ./scripts/run-devnet.sh full-feature
sleep 120
kurtosis service logs eth2030-devnet spamoor 2>&1 | grep -E "error|failed"
# clean = no output

Known-good state checklist (2-minute run)

• cast bn > 40 (30+ blocks in 2 min at 2s slots)
• spamoor: N total tx, N tx confirmed every block, zero errors
• EL logs: zero error / panic lines
• CL logs: (verified) every slot, only Low peer count warnings
• assertoor: check_clients_are_healthy and check_execution_sync_status passing
• finalized_epoch: 0 is OK under 5 minutes

Metrics: reading EL node memory and health

Discover ports

ENCLAVE=eth2030-devnet

# JSON-RPC (eth_blockNumber, eth_getBlock*, etc.)
EL_RPC=$(kurtosis port print $ENCLAVE el-1-geth-lighthouse rpc)

# Prometheus metrics (ETH2030_* counters + docker RSS)
EL_METRICS=$(kurtosis port print $ENCLAVE el-1-geth-lighthouse metrics)
# Output format: "http://127.0.0.1:<port>"

echo "RPC:     http://$EL_RPC"
echo "Metrics: $EL_METRICS/metrics"

Available ETH2030 metrics

# Dump all ETH2030_ metrics
curl -s "$EL_METRICS/metrics" | grep "^ETH2030_"

Memory cost: two complementary views

1 — Docker container RSS (OS perspective, includes all mapped memory):

# Instantaneous RSS for both EL nodes
docker stats --no-stream --format "{{.Name}}\t{{.MemUsage}}" \
  | grep -E "el-[12]-geth"

# Example output:
# el-1-geth-lighthouse--<id>   58.6MiB / 31.26GiB
# el-2-geth-lighthouse--<id>   59.8MiB / 31.26GiB

2 — Go runtime heap (if go_memstats_* are exposed):

curl -s "$EL_METRICS/metrics" \
  | grep -E "^go_memstats_(alloc_bytes|heap_alloc|heap_inuse|heap_sys) "

3 — Combined one-liner for repeated sampling:

EL_METRICS=http://127.0.0.1:32928   # replace with actual port

watch -n 60 'echo "--- $(date -u) ---" && \
  docker stats --no-stream --format "{{.Name}}: {{.MemUsage}}" | grep el-1 && \
  curl -s $EL_METRICS/metrics | grep -E "^ETH2030_(chain_blocks_inserted|chain_height|evm_executions|rpc_requests) "'

Block throughput and health spot-check

EL_RPC=127.0.0.1:32925      # replace with actual port
EL_METRICS=http://127.0.0.1:32928

# 1. Current chain head
cast bn -r http://$EL_RPC

# 2. Blocks processed vs head — mismatch indicates orphaned blocks
BLK_INS=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_chain_blocks_inserted /{print $2}')
CHAIN_HT=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_chain_height /{print $2}')
echo "blocks_inserted=$BLK_INS  chain_height=$CHAIN_HT"

# 3. RPC error rate (high rate = client compatibility issue)
RPC_REQ=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_rpc_requests /{print $2}')
RPC_ERR=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_rpc_errors /{print $2}')
echo "rpc_requests=$RPC_REQ  rpc_errors=$RPC_ERR"

# 4. EVM throughput
EVM_EX=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_evm_executions /{print $2}')
EVM_GAS=$(curl -s $EL_METRICS/metrics | awk '/^ETH2030_evm_gas_used /{print $2}')
echo "evm_executions=$EVM_EX  evm_gas_used=$EVM_GAS"

120-minute soak test with memory logging

Use the pre-built monitor script (requires a running devnet):

# After booting the devnet, start the monitor in background:
/project/eth2030/docs/plans/memory/monitor.sh &

# It writes every 60s to:
#   docs/plans/memory/memory_log.csv     — per-minute CSV
#   docs/plans/memory/monitor.log        — progress log
#
# After 120min it collects service logs and writes:
#   docs/plans/memory/error_report.md    — error summary
#   docs/plans/memory/el-1.log, cl-1.log, spamoor.log, ...

CSV columns: minute, timestamp_utc, block, el1_docker_rss_raw, el1_docker_rss_mb, el2_docker_rss_raw, el2_docker_rss_mb, el1_blocks_inserted, el1_chain_height, el1_chain_reorgs, el1_engine_new_payload, el1_engine_fcu, el1_evm_executions, el1_evm_gas_used, el1_rpc_requests, el1_rpc_errors, el2_blocks_inserted, el2_evm_executions

Memory leak diagnosis

If RSS grows unexpectedly, check these bounded caches (all tunable via CLI flags):

# Reduce cache sizes to diagnose memory source:
docker run eth2030:local --cache.state-snapshots=1 --cache.block=64 ...

# Expected growth pattern after fixes:
# - ~0.9 MB/s per EL node under storagespam scenario (unavoidable: state grows)
# - Plateaus once block cache fills (~256 blocks × block_size)
# - Should NOT grow unboundedly (would indicate a new leak)