spirent-dut-bringup

name: spirent-dut-bringup description: End-to-end workflow for bringing up Spirent emulated traffic and any control-plane / data-plane feature on a DUT through a multi-vendor L2 fabric (Arista trunk → DNAAS bridge-domain → DNOS DUT, often with QinQ), with structured triage when it doesn't work and discovery patterns for finding the right syntax on either side. Use when the user asks to send anything from Spirent toward a DUT, configure devices/streams/protocols on Spirent, push matching config to a DUT to interoperate with Spirent, scale a feature to N sessions/adjacencies/sub-interfaces, or debug why Spirent traffic / control-plane isn't reaching the DUT (no MAC learning, no ARP/ND reply, adjacency stuck, capability negotiation incomplete, etc.). Covers discovery techniques (how to find the right STC class+attribute names, how to find the right DNOS syntax) and the lab's specific gotchas (Arista global-vlan-vs-trunk-allowed, DNAAS BD QinQ inner-tag transparency, virtual-chassis capture broken, Spirent raw-stream etherType chain, control-plane "session up but feature not negotiated" patterns).

spirent-dut-bringup — Spirent → DUT lab workflow

When to use this skill

Trigger phrases / symptoms (any feature):

• "Configure a device on Spirent in vlan X" / "send traffic from Spirent"
• "Set up between Spirent and " — applies to any feature the DUT speaks (routing protocols, MPLS, segment-routing, VRF, EVPN, multicast, OAM, SyncE, anything Spirent emulates)
• "Bring up N <sessions / adjacencies / sub-interfaces>"
• "Spirent ARP/ND fails / no reply"
• "Why doesn't learn the spirent MAC?"
• "Session is up but the feature shows <not negotiated / 0 prefixes / no neighbors / down>"
• A lab path through Spirent → arista → DNAAS-LEAF/SPINE BD → DNOS DUT

If the symptom is purely DUT-side and unrelated to Spirent reachability, prefer debug-routing / debug-datapath / debug-infra instead.

The lab's L2 path (typical shape)

Spirent chassis port  M/N
    │  (one or more VLAN tags)
    ▼
arista trunk port  (mode trunk, allowed vlan X)        ← REQUIRES `vlan X` global!
    │
    ▼
DNAAS-LEAF / SPINE / LEAF chain of <phy>.X sub-IFs
    each sub-IF: l2-service enabled, vlan-id X
    each leaf/spine has a BD instance binding its
    two sub-IFs together. Outer tag X is stripped on
    ingress and re-added on egress; inner tags ride
    through transparently.
    │
    ▼
DUT (DNOS) <phy>[.Y] sub-interface
    (vlan-tags outer-tag X [inner-tag Y], your L3, your MAC)

Concrete details (which Arista, which port, which DNAAS leaf, which DUT sub-IF) come from show interfaces description on the Arista, LLDP on the DNAAS leaves, and network-mapper get_device_lldp / get_dnaas_lldp for the rest of the path. Do this discovery first; don't hard-code anything.

Two facts that drive most decisions:

1. The BD strips and re-adds the outer tag (the one matching its vlan-id). Anything else in the frame — extra VLAN tags, custom ethertypes, MPLS, IP — passes through opaquely.
2. The DUT's sub-interface defines the L2 boundary that Spirent must match. Read its Encapsulation: line first; everything else (IP, protocol enable) flows from getting the encap right.

Workflow (works for any feature)

Phase 1 — Session sanity:
  [ ] Confirm Spirent session exists + its port reservation
  [ ] Confirm physical link UP (LinkStatus on the active phy)
  [ ] Read the DUT-side interface to know what encap to match

Phase 2 — Configure Spirent emulated device(s):
  [ ] Build JSON spec for `python3 -m spirent_mcp.configure`
  [ ] Match the DUT's exact encap (single tag vs QinQ, TPID, etc.)
  [ ] --dry-run first, then real apply
  [ ] **Apply incrementally** — one device at a time (or small batches of 2-3)
      with a brief pause between calls. Large single-transaction applies
      (10+ devices + streams in one shot) can wedge the STC REST API.
      See gotcha #14.
  [ ] If the feature needs control-plane (anything beyond raw frames):
      add the relevant protocol block to the device spec

Phase 3 — L2 reachability:
  [ ] Run ArpNdStartCommand on the device, expect RESOLVE_DONE for v4
  [ ] If RESOLVE_FAILED → use the "L2 debug walk" below
      (do this BEFORE trying to bring up any L3+ feature)

Phase 4 — DUT-side feature config:
  [ ] Read the existing config slice with get_device_config(section=…)
  [ ] Find correct syntax (see "How to find the right DNOS syntax" below)
  [ ] validate_config on a single-unit probe FIRST
  [ ] Scale to full config, validate again, then execute_config
  [ ] On Spirent, DevicesStartAllCommand to arm any protocol blocks

Phase 5 — Validate live state:
  [ ] DUT show commands confirm the feature behavior
  [ ] Spirent's RouterState / per-feature state is ESTABLISHED / Up
  [ ] If a session is "up" but the feature shows "not negotiated /
      not exchanged / 0 things received" → the data-plane or feature-
      specific exchange isn't being driven; see "Session up, feature
      half-baked" below

Tools

L2 debug walk (when ARP/ND fails)

The order matters — go from the closest-known-good point outward:

1. Spirent port physical: physicalport1.LinkStatus = Up and LineSpeedStatus matches the peer's forced speed. If the peer is speed forced 25gfull with no error-correction encoding, Spirent FEC must be off too. Mismatched FEC can yield link-up with silent bit-error drops.
2. Frame format on the wire (the trickiest layer): use Spirent's built-in capture (CaptureStartCommand → CaptureStopCommand → PktCount). Note: virtual containerized chassis (host il-auto-containers) often returns PktCount = 0 — capture is broken there. If so, fall back to peer-side counters / SPAN.
3. Arista (or other adjacent switch) ingress:
   • show interfaces Et<X>/<Y> counters — InUcastPkts climbing? Errors zero?
   • show interfaces Et<X>/<Y> counters errors — FCS / Align / Symbol?
   • show mac address-table | include <oui> — learned in the right vlan?
4. Switch vlan database: show vlan — the vlan must exist as a global config object, not just be in switchport trunk allowed vlan X. THE most common silent-drop cause in this lab.
5. DNAAS BD chain — verify the BD members are on the actual DUT-facing port. The DUT-facing physical link may be a member of a bundle-N, in which case the BD must bind bundle-N.<vlan>, NOT the physical sub-IF ge<X>-0/0/<Y>.<vlan>. Two checks:
   • show interfaces ge<X>-0/0/<Y> — look for Bundle-id: <N>. If non-zero, the port is a bundle member and frames go through the bundle, not the physical port directly.
   • show interfaces bundle-<N> — Members Information table must show Member State: active and LACP Port State: active/active (both columns). standby in the partner column means the bundle's other end isn't speaking LACP back -- see gotcha #20.
   • Then show bridge-domain mac-address-table instance <name> — verify the source MAC propagates from one sub-IF to the other.
   • Description fields are NOT a reliable hint about which port faces which device — they reflect ancient cable patches, not current LLDP. Re-query LLDP at the start of every debug session.
6. DUT ingress: show interfaces <bundle-X.Y> (or <phy-X/Y/Z>.<vlan> for non-bundle ports) — uptime, the Encapsulation: line (compare to what Spirent is sending), show arp for the spirent IP. If the DUT's ARP table populates even without your explicit ARP request, the L2 path is alive in at least one direction.

Discovery patterns (the "I don't know the syntax" loop)

Spirent (STC REST)

The STC server's error messages are the documentation. Use them.

1. Find the right class name: try a stc.create('GuessName', under=...). The error lists "similar classes" — use that to converge:

   stc.create('BfdSession', under='router2')
   # 500: similar classes include BfdSessionDetails and BfdSessionResults
   stc.create('BfdRouterConfig', under='router2')   # works
   

2. Find the right attributes: create the object, then stc.get(h) — the response is the full attribute dict with current values. Use that to learn what's settable and what defaults to expect.

3. Find the right enum value: pass a guess; the error lists the valid set:

   Invalid enum value, 'POINT_TO_POINT': should be one of BROADCAST or P2P
   

4. Find the right perform-command name: the error lists similar:

   unable to create unknown command "BgpStartRouterCommand":
       similar commands include BgpRestartRouterCommand,
       BgpAdvertiseRouteCommand, ...
   

5. Find the right argument name for a command: pass any name; the error lists the valid arg set:

   invalid bgpadvertiseroutecommand attribute "routerlist":
       should be ... RouteList ...
   

This iterative-probe loop is faster than searching docs and works for anything in STC.

DNOS (DUT)

Two roughly-equivalent sources of truth, in this order of preference:

1. Golden configs in device_golden_config/ — came from working live commits, so the hierarchy and keyword choices are guaranteed to parse on the DUT version that produced them. Grep '<keyword>' device_golden_config -A 15 finds a working snippet fast.
2. validate_config round-trip — when no golden has the feature you need (or it's new in this DNOS release), paste a minimal single-block probe and let the DUT's commit check parser tell you exactly which token it can't recognize and where in the line. Round-trip is a few seconds.
3. search_cli_docs is keyword-literal and frequently misses compound forms; useful for finding the exact attribute names and ranges once you know the right command, but a poor first stop for "does this feature exist?"

Whatever the source, validate a small probe block first, then scale. Match the hierarchy (iso-network not net, bgp <AS> not separate local-as, level level-2 not level 2, bfd-type single-hop inside the BGP neighbor X bfd block, etc.). The device-setup skill covers the probe-then-scale discipline in detail; read it before pushing DUT config.

"Session up, feature half-baked"

For any control-plane protocol (BGP, LDP, RSVP, BFD, etc.), Spirent's emulated router will reach an "Established" / "Up" state after the peer-to-peer handshake, but feature-specific negotiation often won't complete until the Spirent side actually transmits a feature payload:

• BGP: stays at (NoNeg) for the AFI/SAFI until a route is advertised (BgpIpv4RouteConfig + BgpAdvertiseRouteCommand) or an EoR is sent.
• LDP: peer is up, but no labels until you create label ranges.
• RSVP: hello-up doesn't mean tunnels — you need explicit tunnel objects.
• BFD: session won't form unless the protocol it's tied to is also up AND both sides have BFD enabled with compatible timers.

Pattern: if "session up but no ", look for the Spirent config object that owns and the perform-command that drives it (advertise / start / refresh). The BgpAdvertiseRouteCommand fix is the canonical case (see the "session up but feature column reads (NoNeg) / 0 things" gotcha row).

Common failure modes

This is the gotcha table. Each row is something hit live in this lab.

DNOS show commands quick-ref

DNOS is NOT Junos-style; many show protocols ... forms don't exist. Use these:

DNAAS devices (DNOS in switch role) accept the same commands; only execute_config is blocked on them — use validate_config for commit-check, then commit manually.

Anti-patterns

• Rebuilding Spirent devices instead of restarting them. Once DevicesStartAllCommand has been issued, prefer the protocol's …RestartRouterCommand over delete-and-recreate. Recreating triggers re-numbering of handles (router2 → router7 → router17 → …) and orphan interfaces that fail apply with "duplicate IPv4/MAC" validation errors.
• Pushing DUT config without first probing one block. The device-setup skill's "single-unit probe before scale" rule applies for every feature, not just interfaces. Validate the smallest meaningful slice first.
• Inferring the L2 path is broken from Spirent symptoms alone. The reverse direction (DUT → Spirent) usually works fine through the BD; when ARP/ND fails, it's almost always Spirent → switch (frame format) or the switch's missing global vlan, NOT the BD chain.
• Using one parameter set on Spirent and expecting the other side to adapt. Network-type (P2P vs broadcast), TPID (0x8100 vs 0x88a8), hello/hold timers, capability sets — must match on both sides. When in doubt, read the DUT's live config and mirror.
• Treating "session up" as "feature ready". Always verify the feature-level state (prefix counts, label counts, mroute counts, etc.), not just the session state.

Reference

• Configure script source: spirent_mcp/configure/cli.py — JSON spec schema in the docstring at the top. Builder code: spirent_mcp/configure/{devices,streams,protocols}.py. To add a new protocol, write create_<name> in protocols.py with the same signature as create_bgp and append to PROTOCOL_BUILDERS.
• Inspect script source: spirent_mcp/inspect/cli.py — read-only; safe to run anytime to see current session state.
• Reach script source: spirent_mcp/reach/cli.py — ICMP + TCP + HTTP; first thing to run when REST won't talk. NOTE: its "Verdict" message can be misleading (it expects a /cgi-bin/ UI). The actual REST endpoint is /stcapi/sessions — if that returns HTTP 200 with a JSON list, the Lab Server is fine regardless of what the verdict says.
• Defaults: spirent_mcp/common/defaults.py — DEFAULT_LAB_SERVER, DEFAULT_SESSION_ID. The defaults can drift from reality; the canonical way to find the current session ID is to run python3 -m spirent_mcp.inspect with no --session-id and let the not-found error print the live list.
• Worked examples (concrete protocol configs from past bringups): see examples.md — keep this updated when a new feature has been brought up successfully so future runs can copy a working config instead of re-discovering syntax.
• Configure-dnos-device skill — read it before modifying DUT config; covers the probe-then-scale discipline and DNOS-specific CLI gotchas in detail.
• STC error messages ARE the documentation — every "Unknown" / "Invalid" / "read-only" error includes the valid alternatives. Use the Discovery patterns section to converge.