a7-plugin-fault-injection

name: a7-plugin-fault-injection description: >- Skill for configuring the API7 Enterprise Edition fault-injection plugin via the a7 CLI. Covers injecting delays and HTTP aborts for chaos engineering, percentage-based sampling, conditional injection via vars expressions, custom response headers and body with Nginx variable interpolation. version: "1.0.0" author: API7.ai Contributors license: Apache-2.0 metadata: category: plugin apisix_version: ">=3.0.0" plugin_name: fault-injection a7_commands: - a7 route create - a7 route update - a7 config sync

a7-plugin-fault-injection

Overview

The fault-injection plugin in API7 Enterprise Edition (API7 EE) injects faults — delays and HTTP aborts — into requests for chaos engineering and resiliency testing. It runs in the rewrite phase with priority 11000 (very early), meaning it executes before most other plugins including authentication and rate limiting.

Execution order: delay first → abort second. If abort fires, subsequent plugins do NOT execute.

When to Use

• Chaos engineering: simulate upstream failures and slowdowns.
• Resiliency testing: verify timeout handling and circuit breakers.
• Load testing: add artificial latency to measure degradation.
• Canary fault testing: inject faults for specific users or conditions.

Plugin Configuration Reference

At least one of abort or delay must be specified.

abort Object

delay Object

Vars Expression Syntax

The vars field uses logical expressions for conditional fault injection.

Structure

[
  [["condition1a"], ["condition1b"]],  // AND group 1
  [["condition2a"]]                     // AND group 2
]
// Groups joined by OR — first matching group triggers the fault

Variable Access

Operators

Step-by-Step Examples

1. Fixed Delay (3 Seconds)

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "delay-test",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "delay": {
        "duration": 3
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

2. Percentage-Based Abort (50% Return 503)

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "abort-test",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 503,
        "body": "Service temporarily unavailable",
        "percentage": 50
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

3. Conditional Abort Based on Query Parameter

Only abort when ?name=jack:

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "conditional-abort",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 403,
        "body": "Fault Injection!\n",
        "vars": [
          [["arg_name", "==", "jack"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

4. Complex Conditional Logic (AND/OR)

Abort when (name=jack AND age≥18) OR (has api-key header):

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "complex-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 403,
        "body": "Fault Injection!\n",
        "vars": [
          [
            ["arg_name", "==", "jack"],
            ["arg_age", "!", "<", 18]
          ],
          [
            ["http_apikey", "==", "api-key"]
          ]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

5. Custom Headers with Nginx Variables

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "headers-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 200,
        "body": "{\"uri\": \"$uri\"}",
        "headers": {
          "X-Fault-Injected": "true",
          "X-Request-URI": "$uri"
        }
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

6. Canary Fault Testing

Only users with X-Canary: true header experience 10% fault rate:

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "canary-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "abort": {
        "http_status": 500,
        "percentage": 10,
        "vars": [
          [["http_x_canary", "==", "true"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

7. Combined Delay + Abort with Different Conditions

a7 route create --gateway-group default -f - <<'EOF'
{
  "id": "combined-fault",
  "uri": "/api/*",
  "plugins": {
    "fault-injection": {
      "delay": {
        "duration": 2,
        "vars": [
          [["http_x_slow", "==", "true"]]
        ]
      },
      "abort": {
        "http_status": 503,
        "vars": [
          [["http_x_fail", "==", "true"]]
        ]
      }
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": [{"host": "backend", "port": 8080, "weight": 1}]
  }
}
EOF

Config Sync Example

version: "1"
gateway_group: default
routes:
  - id: fault-injection-demo
    uri: /api/*
    plugins:
      fault-injection:
        delay:
          duration: 1
          percentage: 25
        abort:
          http_status: 503
          body: "Service unavailable"
          percentage: 5
    upstream:
      type: roundrobin
      nodes:
        - host: backend
          port: 8080
          weight: 1

Execution Behavior

1. Delay evaluated first: if vars match and percentage sampled → sleep(duration).
2. Abort evaluated second: if vars match and percentage sampled → return immediately.
3. Percentage sampling: math.random(1, 100) <= percentage.
4. When abort fires: subsequent plugins (auth, rate limiting) are skipped.

Plugin Priority Context

Priority 11000 means fault-injection runs very early:

• ✅ Tracing plugins (zipkin, skywalking) capture faults.
• ❌ Rate limiting won't prevent faults.
• ❌ Authentication won't block faults.

To apply faults only to authenticated users, use vars to check auth-related variables or headers.

Troubleshooting