data-pipeline

name: data-pipeline description: Build and manage input/output pipelines for the Discovery Engine

Data Pipeline Skill

Use this skill when building or maintaining the Input/Output Pipelines for the Discovery Engine. This manages the "Feed" of strategies to test and the "Storage" of results.

Context

The Core Engine needs a steady stream of test definitions (strategies.json) and a place to store winners (results.db).

1. Input Schema (strategies.json)

Define a robust JSON schema for a "Test Job":

[
  {
    "id": "job_001",
    "trigger": {"name": "RSI", "params": [14], "buffer": 0, "rule": ">70"},
    "confirm": {"name": "MovAvg", "params": [50], "buffer": 0, "rule": "slope_up"}
  }
]

2. StrategyLoader

Script to read/validate the JSON:

• Include a "Permutation Generator"
• Function that takes RSI and [14, 21, 28] and expands it into 3 separate test job keys

3. ResultLog

Technology: SQLite (via Python sqlite3)

Schema:

• Table: discovery_results
• Columns: id, timestamp, trigger_name, confirm_name, win_rate, edge_score, trade_count, params_json

Function: save_result(result_dict) checking for duplicates

4. Key Files

Constraints

• Robustness: Malformed JSON should not crash the pipeline; log error and skip
• Separation: Do not import bridge or logic. This is purely Data IO