experiment-execution

Phase 4a is deliberately QUICK — it should take a fraction of the effort of full execution. The goal is insight, not completeness.

Phase 4a for Type M — Baseline Probing + Initial Assessment

Goal: Understand the actual problem landscape before committing to full-scale execution.

NOTE: Baselines run here count toward Phase 4b Stage B — they will NOT be re-run. Phase 4a is a subset of the full execution, not a separate exercise.

1. Set up minimal infrastructure — data loading, evaluation function, seed control
2. Run 1-2 strongest baselines on 1 representative dataset, full execution (not toy run)
3. Analyze baseline results:
   • What's the actual performance level? Does it match literature?
   • Where do baselines fail? (error analysis, failure cases, edge cases)
   • Is there a systematic failure pattern? Is it what we predicted in Phase 3?
4. Quick prototype of our method (optional but recommended) — run a minimal version to see if the core idea shows promise
5. Quick literature check if findings are surprising

Output: docs/05_execution/phase4a-exploration-report.md

Phase 4a Exploration Report (Type M):
═════════════════════════════════════

Baselines run: [list with results]

Error analysis findings:
  [describe failure patterns, edge cases, systematic issues]

Predicted failure pattern (from Phase 3): [X]
Actual failure pattern: [Y]
Alignment: [matched / partially matched / different]

Quick prototype results (if run): [summary]

Issues or surprises: [list anything unexpected]

Phase 4a for Type D — Preliminary Analysis Run

Goal: Run the core analysis pipeline once on real data to see what the actual results look like, before committing to the full analysis plan.

NOTE: "Exploratory analysis" means different things in different fields. It is NOT limited to clustering and dimensionality reduction. The point is to execute the main planned analysis at a basic level and examine the output.

1. Preprocessing — quality control, normalization, whatever the domain requires
2. Run the core analysis pipeline once — whatever the Phase 3 plan calls for as the main analysis (could be differential expression, could be model fitting, could be statistical tests, could be simulation — depends on the field)
3. Examine the results:
   • Do the results make sense given domain knowledge?
   • Are the expected signals present?
   • Is there something unexpected or interesting?
   • Is the data quality sufficient for the planned analysis?
4. Assess the analysis plan:
   • Does the Phase 3 plan still make sense given what we see?
   • Are some planned analyses unnecessary? Are there gaps?

Output: docs/05_execution/phase4a-exploration-report.md

Phase 4a Exploration Report (Type D):
═════════════════════════════════════

Data quality: [summary]

Preliminary analysis results:
  [describe key outputs from the initial analysis run]

Results vs expectations:
  Expected: [what Phase 3 predicted]
  Observed: [what actually came out]
  Surprises: [anything unexpected]

Assessment of analysis plan:
  Still appropriate? [yes / needs adjustment]
  Missing analyses? [list]
  Unnecessary analyses? [list]

Phase 4a for Type C — Quick Benchmark Probe

Goal: Run a quick comparison on one benchmark to see if the tool's advantages are real.

1. Run our tool + 1-2 main competitors on 1 representative benchmark
2. Compare: correctness, speed, memory
3. Assess: Does our claimed advantage hold? How big is the gap?
4. Test installation: Does it install cleanly on a fresh environment?

Output: Append to docs/05_execution/phase4a-exploration-report.md

Phase 4a Exploration Report (Type C):
═════════════════════════════════════

Quick benchmark: [name]
Our tool: [correctness, runtime, memory]
Competitor A: [correctness, runtime, memory]
Competitor B: [correctness, runtime, memory]

Claimed advantage: [X]
Actual advantage: [Y]
Gap: [large / moderate / marginal / none]

Installation test: [clean / issues]

Phase 4a — Optional: Theoretical Analysis / Proof Sketch

If the project includes theoretical claims (convergence, complexity, correctness, bounds):

1. Sketch the proof strategy — identify the key lemmas needed
2. Attempt the easiest lemma first — to check if the overall approach is viable
3. Identify blockers — are there steps that require assumptions we can't justify?
4. Assess feasibility — can we prove what we claimed, or do we need to weaken the claim?

This can run in parallel with the computational exploration above.

Output: Append to docs/05_execution/phase4a-exploration-report.md

Theoretical Analysis (preliminary):
  Claim: [what we want to prove]
  Proof strategy: [approach]
  Key lemma attempted: [which one, result]
  Feasibility: [viable / needs weaker claim / blocked by X]

⛔ Phase 4a Decision Point (MANDATORY STOP)

Based on the exploration results, assess the situation and present appropriate options. The most common outcome is local adjustment, not a full phase reset.

Phase 4a exploration complete. Report above.

MY ASSESSMENT: [summarize what was found and what it means for the plan]

YOUR OPTIONS:

1. ✅ PROCEED AS-IS — Plan is confirmed. Continue to Phase 4b.

2. 🔧 LOCAL ADJUSTMENT (most common) — Adjust specific parts of the 
   plan based on what we learned:
   [list specific proposed changes, e.g.:]
   - Add experiment X to address [finding]
   - Modify analysis step Y because [reason]
   - Replace baseline Z with a stronger one
   - Adjust method component W based on error analysis
   These changes are applied directly — no need to redo Phase 3.

3. 🔄 FOCUSED RE-DISCUSSION — Some findings need deeper thought. I'll 
   dispatch the Phase 3 agents (Innovation Advisor / Domain Scientist / etc.)
   for a FOCUSED discussion on the specific issue:
   "[describe the specific issue that needs multi-agent input]"
   This is NOT a full Phase 3 redo — just a targeted consultation.

4. ↩️ RETURN TO PHASE 3 — The method/analysis design has a FUNDAMENTAL 
   problem that can't be fixed with local adjustments.
   [Only suggest this if there's a deep structural issue.]

5. ↩️↩️ RETURN TO PHASE 2 — The research question itself is wrong.
   [Only suggest this in extreme cases — e.g., the data contradicts 
   the premise, or the problem doesn't exist.]

Which option? (You can also propose your own adjustments.)

Guidelines for which option to suggest:

For Option 2 (Local Adjustment):

• Apply the changes directly to Phase 3 outputs (evaluation protocol, analysis plan, etc.)
• Document what changed and why in the exploration report
• Proceed to Phase 4b with the updated plan

For Option 3 (Focused Re-discussion):

• Dispatch the relevant Phase 3 agents with the Phase 4a findings
• Give them a FOCUSED prompt: "Based on these exploration results, should we [specific question]?"
• Agents see the exploration report + the original plan
• Apply their recommendations and proceed to Phase 4b
• This is a LIGHTWEIGHT version of the Phase 3 deliberation — max 2-3 rounds, focused on one issue

For Options 4 and 5 (Full return):

• Save exploration findings in docs/05_execution/phase4a-exploration-report.md
• Return to the appropriate phase via its "Re-entry from Phase 4a" path
• Carry forward all exploration insights

The user can also iterate within Phase 4a: If the user says "try X first" or "run one more experiment before deciding," that's fine — Phase 4a can have multiple rounds of quick exploration before the decision point. The MANDATORY STOP is about getting user input before committing to full-scale Phase 4b, not about limiting exploration.

Phase 4b — Full Execution

Type M Execution Flow (Phase 4b)

Stage A — Build Experiment Framework

Set up the infrastructure before any experiment runs:

1. Data loading pipeline (scripted, deterministic)
2. Evaluation function matching evaluation-protocol.yaml exactly
3. Logging framework (metrics, configs, seeds, timestamps)
4. Reproducibility scaffolding: seed control, environment snapshot, config saving

Use implementer-prompt.md to dispatch coding tasks.

Stage B — Complete Baseline Reproduction

IRON LAW: BASELINES FIRST. NO EXCEPTIONS.

NOTE: If Phase 4a already ran some baselines, carry those results forward — do NOT re-run them. Stage B completes the REMAINING baselines and consolidates all results.

B.1 — Check for pre-computed and Phase 4a results

Before implementing anything, check two sources:

1. evaluation-protocol.yaml — each baseline's pre_computed.status
2. docs/05_execution/phase4a-exploration-report.md — baselines already run in Phase 4a

Report to user:

Baseline execution plan:
  Skip (pre-computed accepted): N baselines
  Re-run (reference only / incompatible / new): M baselines
  Total baselines: N + M

B.2 — Run baselines that need running

For each baseline that needs running:

1. Implement or integrate from official source
2. Run with ALL seeds from evaluation-protocol.yaml
3. Compare with paper-reported values (or reference_only pre-computed values)

B.3 — Consolidate all baseline results

Merge accepted pre-computed results and newly reproduced results into a single table in docs/05_execution/baseline-results.md:

All downstream comparisons (Stage D, Stage E) use this consolidated table.

Stage C — Implement Core Method (+ Optional Theoretical Analysis)

Build incrementally — each component must be independently testable:

1. Implement smallest viable version first
2. Unit test each module
3. Follow reproducibility-driven-research: fix seeds, log environment, script everything
4. Commit after each working component

If the project includes theoretical claims (convergence, complexity, correctness):

Complete the formal proofs alongside or before full experimental validation:

1. State theorems formally — precise conditions, claims, and notation
2. Prove each theorem — show complete derivations, not hand-waving
3. Verify assumptions — ensure the conditions required by the proof hold in practice
4. Connect theory to experiments — the experiments should validate what the theory predicts

Write proofs to docs/05_execution/theoretical-analysis.md (or .tex for LaTeX).

Common proof types in computational research:

If a proof attempt FAILS (can't prove the desired result):

• Document what went wrong — is the claim too strong?
• Consider weakening the theorem (e.g., weaker conditions, looser bound)
• Discuss with user: proceed with weaker theoretical guarantee, or redesign the method?

Stage D — Initial Experiments + Iteration

=== Iteration Round N Report ===
Hypothesis: [what was wrong]
Change: [what was modified]
Result:
  This round:  XX.X ± Y.Y (primary metric)
  Last round:  XX.X ± Y.Y
  Best baseline: ZZ.Z ± W.W
  Gap to best baseline: ±N.N%
Status: [improving / stagnating / regressing]
Recommendation: [continue iterating / ready for full-scale / consider pivot]

Call Task tool with:
  description: "[short task name]"
  prompt: |
    [Contents of the filled-in prompt template]
  subagent_type: "generalPurpose"

Call Task tool with:
  description: "Run baseline: GraphSAGE"
  prompt: |
    [Filled implementer-prompt.md for GraphSAGE]
  subagent_type: "generalPurpose"

Call Task tool with:
  description: "Run baseline: GAT"
  prompt: |
    [Filled implementer-prompt.md for GAT]
  subagent_type: "generalPurpose"

Call Task tool with:
  description: "Run baseline: GIN"
  prompt: |
    [Filled implementer-prompt.md for GIN]
  subagent_type: "generalPurpose"

Phase 4 complete. Results summary:
  - [list key results, metrics, figures generated]
  - [list any issues or limitations found]

Shall I proceed to Phase 5 (Results Integration)?
This will include a multi-agent discussion panel to stress-test
the results before paper writing.