evaluation

name: evaluation description: Complete reference for the config-first model evaluation system. Covers the Evaluator CLI, assertion-driven YAML scenarios, response views, backend configuration, presets, scoring, LLM-as-judge, model comparison, and HuggingFace integration. Use when evaluating models, writing test prompts, comparing training runs, or interpreting eval results. This skill is about USING the evaluation system via CLI and YAML. allowed-tools: Read, Bash, Write, Grep, Glob

Model Evaluation

Config-first evaluation framework for testing model responses against YAML-defined correctness assertions.

The evaluator does not hardcode a specific tool family, manager id, wrapper name, or behavior rule as correctness. Scenarios define the prompt and the acceptable response shape directly under correct.

Quick Reference

Status System

Schema/structural validation may still be reported for debugging, but it is not the source of task correctness. Correctness belongs in scenario YAML.

Key Directories

• Evaluator/ - Core evaluation code
• Evaluator/config/scenarios/ - YAML test scenarios
• Evaluator/config/tool_schema.yaml - Current CLI wrapper/tool schema metadata
• Evaluator/config/rubrics/ - LLM-as-judge rubrics
• Evaluator/results/ - Evaluation output JSON and Markdown

Progressive Reference

Active Scenario Pattern

Every test should define what counts as correct:

tests:
  - id: storage_copy_runbook
    question: Copy the incident runbook into a template file.
    tags: [storageManager, single-tool]
    system: |
      <session_context>
      sessionId: "session_eval"
      workspaceId: "ws_eval"
      </session_context>
    correct:
      any:
        - name: copy_cli
          assertions:
            - type: jsonpath_equals
              path: $.tool_calls[0].name
              value: useTools
            - type: jsonpath_regex
              path: $.tool_calls[0].arguments.tool
              pattern: '^storage copy\b(?=.*Incident-Response\.md)(?=.*Incident-Response-Template\.md)'

Use correct.any for multiple valid answers, such as command by id or by name. Use correct.all or nested all/any/not assertions for stricter structures.

Response View

Assertions query a generic response view. This is syntax normalization only:

• $.raw preserves the raw assistant response.
• $.content is assistant text.
• $.content_json is parsed JSON content when content is JSON.
• $.tool_calls is a normalized list of emitted tool calls.
• OpenAI-style function.arguments JSON strings are parsed into objects.
• Plain text blocks like tool_call: useTools plus arguments: {...} are parsed into the same view.

The response view must not map CLI commands to old manager tool ids or decide correctness. Scenario YAML decides what is correct.

Tips

• Keep all task-specific expectations in YAML under correct.
• Do not add evaluator code for a specific tool, wrapper, or use case.
• Prefer regex or JSONPath assertions for tool CLI commands, because shell quoting and argument order can vary.
• If a schema allows equivalent forms, represent them as separate correct.any paths.
• Use --limit and --tags for fast iteration.
• Use --validate-context only when the scenario includes context fields that should be structurally checked.
• Use --env-backend local or e2b only when you need runtime execution checks beyond response correctness.