snakemake-signal-isolation

name: snakemake-signal-isolation description: "Adding signal isolation as a Snakemake pipeline rule for end-to-end automation. Trigger: signal isolation rule, pipeline integration, batch completion sentinel, autofluorescence subtraction as workflow step." author: KINTSUGI Team date: 2026-02-24

Signal Isolation as Snakemake Rule

Experiment Overview

Context

Signal isolation (autofluorescence subtraction) was a separate CLI command (kintsugi workflow isolate run) that had to be manually invoked after the Snakemake pipeline finished registration. This broke end-to-end automation — kintsugi workflow batch only ran through registration, then signal isolation required a second pass. The fix was to make it a proper Snakemake rule like every other pipeline stage.

Verified Workflow

Pipeline After Change

stitch → decon → edf (per-cycle) → registration (aggregate) → signal_isolation (aggregate) → 5 QC rules

Rule Design: CPU-Only Aggregate (Like Registration)

Signal isolation is an aggregate rule — no {cycle} wildcard, processes all channels at once. CPU-only (numpy/scipy, no GPU needed). Reuses _QC_CPU_ASSIGN for SLURM partition routing (same pattern as existing QC rules).

rule signal_isolation:
    input:
        reg_done=f"{PROJECT}/data/processed/registered/.snakemake_complete",
    output:
        sentinel=f"{PROJECT}/data/processed/signal_isolated/.snakemake_complete",
    resources:
        mem_mb=RES.get("mem_signal_isolation", 32000),
        runtime=RES.get("time_signal_isolation", 120),
        slurm_partition=_QC_CPU_ASSIGN["partition"],
        slurm_account=_QC_CPU_ASSIGN["account"],
        gres="",
        cpus_per_task=RES.get("cpus_signal_isolation", 4),
    script: "scripts/signal_isolation.py"

Wrapper Script Pattern (scripts/signal_isolation.py)

Follows the registration.py pattern:

1. Read snakemake.params and snakemake.config
2. Setup 3-tier sys.path (project → kintsugi notebooks → kintsugi root)
3. Import log_utils from scriptdir
4. Read signal_isolation config section
5. Skip-existing: If manifest exists and all marker TIFs present → write sentinel and exit
6. Recipe auto-discovery: Config recipe_dir → standard search paths (Processing_parameters/)
7. Call process_batch() from kintsugi.signal.batch
8. Write sentinel with metadata (method counts, quality, timing)

Config Section (config.yaml)

signal_isolation:
  method: auto          # auto, global, or weighted
  tissue_type: spleen   # Auto-detected from project name via parse_tissue_type()
  tile_smooth_sigma: 0.0
  recipe_dir: ""        # Empty = auto-discover in standard paths
  learn: true
  force: false

resources:
  mem_signal_isolation: 32000
  time_signal_isolation: 120
  cpus_signal_isolation: 4
  time_qc_signal_isolation: 60

Tissue Type Auto-Detection

generate_workflow_config() calls parse_tissue_type(project_dir.name, project_dir) from batch_multi.py:

• _SP_ or spleen → "spleen", _LN_ or lymph-node → "lymph_node", _TH_ or thymus → "thymus"
• Falls back to experiment.json name field, then "unknown"

QC Integration

qc_signal_isolation dispatches to generate_qc_pages() from signal/isolation_qc.py via the existing qc_report.py dispatcher. No cache file needed — signal isolation QC is self-contained (reads manifest + registered + isolated images directly).

Batch Completion Sentinel Change

The batch completion sentinel changed from registered/.snakemake_complete to signal_isolated/.snakemake_complete. This affects:

• _discover_batch_projects() — projects with only registration are now "eligible" (not "completed")
• _detect_project_stage() — signal_isolated is the new highest stage
• rule all target — changed from registered to signal_isolated
• all_qc_sentinels() — 5 sentinels instead of 4

Files Modified

Failed Attempts (Critical)

Key Insights

• CPU-only rules use _QC_CPU_ASSIGN — same partition routing pattern as QC rules; no need for lambda resources or GPU assignment
• Aggregate rules follow registration pattern — static resources, f-string outputs, no {cycle} wildcard
• Wrapper scripts follow 3-tier sys.path — project notebooks → kintsugi notebooks → kintsugi root
• Sentinel change ripples through — batch eligibility, stage detection, rule all, QC sentinels, cleanup gate ALL need updating
• Recipe auto-discovery makes the rule zero-config — searches standard paths, falls back to auto-analysis
• Tissue type auto-detection in generate_workflow_config() means users don't need to manually configure tissue type
• Re-config propagates via workflow config — always overwrites Snakefile + profiles; adds new scripts per-file. BUT existing scripts are NOT updated — must bulk-copy after source changes

Verified On

• 35 projects re-configured via kintsugi workflow config . (Feb 24, 2026)
• All tissue types correctly auto-detected (spleen, lymph_node, thymus)
• 92/92 tests pass (26 workflow batch + 66 batch signal isolation)
• 1901CC3C completed full pipeline through Snakemake (SI + QC)
• 1901CC2A: SI succeeded, QC failed due to stale qc_report.py in project (missing SI handler). Fixed by bulk-copying updated script (Feb 25, 2026)
• 25 batch-processed projects validated and promoted via scripts/create_si_sentinels.py (see sentinel-validation-promotion skill)

References

• snakemake-workflow-architecture skill — base workflow design
• batch-signal-isolation skill — process_batch() internals
• claude-md-context-management skill — CLAUDE.md size management