sdrf-techrefine

name: sdrf:techrefine description: Use when the user wants to refine or verify SDRF technical metadata (instrument, tolerances, modifications, DDA/DIA) using raw MS file analysis via techsdrf. user-invocable: true argument-hint: "[PXD accession or SDRF file path]"

SDRF Technical Metadata Refinement Workflow

You are guiding the user through refining SDRF technical metadata using techsdrf — a CLI tool that downloads raw MS files, analyzes them with pyopenms, and auto-detects instrument parameters, mass tolerances, PTMs, and DDA/DIA mode.

This workflow verifies and corrects technical columns that are often filled manually from PRIDE metadata or publications, which may be incomplete or inaccurate.

Step 1: Check Prerequisites

1.1 Verify techsdrf is installed

techsdrf --version
# If not installed:
pip install techsdrf

1.2 Verify raw file converters (as needed)

• Thermo .raw files: ThermoRawFileParser is required

  conda install -c bioconda thermorawfileparser
  

• Bruker .d / SCIEX .wiff files: msconvert (ProteoWizard) is required

  # Check availability:
  msconvert --help
  

If converters are missing, inform the user which file types cannot be processed and suggest installation commands. techsdrf will skip files it cannot convert.

Step 2: Choose Refinement Mode

Present the three modes and help the user choose:

Mode A — Refine from PRIDE (recommended for PXD datasets)

Downloads raw files from PRIDE, analyzes them, and refines the SDRF.

techsdrf refine -p PXD###### -s input.sdrf.tsv -o refined.sdrf.tsv -v

Use when: The user has a PXD accession and an SDRF file to refine.

Mode B — Refine from local files

Points to a local directory of raw files instead of downloading from PRIDE.

techsdrf refine -d /path/to/raw/files -s input.sdrf.tsv -o refined.sdrf.tsv -v

Use when: The user already has raw files locally.

Mode C — Info only (inspect without refining)

Inspects the current SDRF technical metadata without downloading or analyzing files.

techsdrf info -s input.sdrf.tsv

Use when: The user wants a quick check of what technical metadata is declared.

Step 3: Configure Analysis

Help the user set the right options for their analysis:

3.1 Number of files to analyze

• -n 3 (default): Analyze 3 representative files — fast, usually sufficient
• -n 5 or -n 10: More files for higher confidence
• -a: Analyze ALL files — thorough but slow; use for heterogeneous datasets where different files may have different instruments or acquisition methods

3.2 File type filter

• -t raw: Thermo .raw files only
• -t d: Bruker .d directories only
• -t wiff or -t wiff2: SCIEX files only
• If not specified, techsdrf processes all supported file types

3.3 Tolerance unit

• -u ppm (default): Report mass tolerances in parts per million
• -u Da: Report mass tolerances in Daltons

3.4 PTM detection

• By default, techsdrf performs three tiers of PTM detection
• --skip-ptm: Skip PTM detection (faster, use if only instrument/tolerance info needed)

3.5 Keep downloaded files

• --keep-files: Retain downloaded raw and converted mzML files after analysis
• By default, files are cleaned up to save disk space

Step 4: Run techsdrf

4.1 Present the full command

Assemble the command with all chosen options and present it to the user:

techsdrf refine \
  -p PXD###### \
  -s input.sdrf.tsv \
  -n 5 \
  -o refined.sdrf.tsv \
  -v

4.2 Explain what happens

The techsdrf pipeline runs these stages:

1. Download — Fetches N raw files from PRIDE (or reads local files)
2. Convert — Converts raw files to mzML using ThermoRawFileParser or msconvert
3. Analyze — Parses mzML with pyopenms to extract instrument metadata, scan characteristics, and spectra
4. Detect — Determines instrument model, acquisition mode, fragmentation, tolerances, labels, and PTMs
5. Compare — Compares detected parameters against declared values in the SDRF
6. Refine — Produces a refined SDRF with corrected/added technical metadata

4.3 The user runs the command

The user executes the command in their terminal. techsdrf produces:

• Console output with detected parameters and comparison results
• A refined SDRF file (if -o was specified)

Step 5: Interpret Results

Parse the techsdrf output and explain each detected parameter:

5.1 Instrument detection

What to check: Does the detected model match what's in the SDRF? PRIDE often has generic names (e.g., "Q Exactive") while raw files have the specific model (e.g., "Q Exactive HF-X").

5.2 Acquisition mode

What to check: Is the acquisition method in the SDRF correct? Misclassification of DDA as DIA (or vice versa) affects which analysis pipelines can be used.

5.3 Fragmentation

What to check: Does the SDRF declare the correct fragmentation method? Multiple methods may be used (e.g., HCD + EThcD for different scans).

5.4 Mass tolerances

techsdrf uses two independent methods to estimate tolerances. Both are reported for comparison. What to check: Do the detected tolerances match the SDRF? Papers often round tolerances or report search engine settings instead of instrument-level tolerances.

5.5 Labels

What to check: Does the labeling in the SDRF match what's actually in the spectra? This catches cases where TMT is declared but the data is actually label-free (or vice versa).

5.6 PTM detection (three tiers)

What to check: Are all detected PTMs declared as comment[modification parameters] in the SDRF? Are there declared modifications that were NOT detected (may indicate incorrect annotation)?

5.7 Comparison summary

techsdrf reports each parameter with a status:

• MATCH — SDRF value matches detected value
• MISMATCH — SDRF value differs from detected value (flag for correction)
• MISSING_SDRF — Detected in raw data but not declared in SDRF (flag for addition)
• IMPROVED — Detected value is more specific than SDRF value (e.g., "Q Exactive" → "Q Exactive HF")

List all MISMATCH and MISSING_SDRF items with recommended fixes.

Step 6: Apply Refinements

6.1 If techsdrf produced a refined SDRF file

Present the differences between original and refined SDRF:

CHANGED: comment[instrument] — "Q Exactive" → "Q Exactive HF-X" (AC=MS:1002877;NT=Q Exactive HF-X)
CHANGED: comment[precursor mass tolerance] — "20 ppm" → "10 ppm"
ADDED:   comment[dissociation method] — "HCD" (was empty)
CHANGED: comment[modification parameters] col 3 — REMOVED Phospho (not detected in spectra)

6.2 Let user approve/reject each change

For each changed column:

• Show the old value and the new value
• Explain WHY techsdrf made this change (what evidence from raw data)
• Let the user approve or reject the individual change

6.3 If the user is working on an in-memory SDRF from /sdrf:annotate

Instead of a file diff, show the corrected values to paste into the SDRF:

Update these technical columns based on raw file analysis:
  comment[instrument]: AC=MS:1002877;NT=Q Exactive HF-X
  comment[precursor mass tolerance]: 10 ppm
  comment[fragment mass tolerance]: 20 ppm
  comment[dissociation method]: NT=HCD;AC=MS:1000422

Step 7: Post-Refinement

7.1 Update spec to latest version

Before validating, ensure the SDRF specification and templates are up to date:

git submodule update --remote --recursive

This pulls the latest column definitions (spec/sdrf-proteomics/TERMS.tsv) and template versions (spec/sdrf-proteomics/sdrf-templates/templates.yaml). Validation against outdated templates may miss new required columns or accept deprecated formats.

7.2 Validate with sdrf-pipelines

Always run programmatic validation on the refined SDRF before presenting results to the user:

parse_sdrf validate-sdrf \
  --sdrf_file refined.sdrf.tsv \
  --template <template1> \
  --template <template2>

Detect templates from comment[sdrf template] columns in the SDRF. If parse_sdrf is not installed, tell the user: pip install sdrf-pipelines

If validation finds errors introduced by the refinement, fix them and re-run until validation passes (or only warnings remain).

7.3 Contribute

If this is a ProteomeXchange dataset:

Your refined SDRF has verified technical metadata from actual raw file analysis.
Run /sdrf:contribute to submit this annotation to the community repository.

What techsdrf Detects

Important Rules

• techsdrf requires actual raw MS files — it cannot refine from metadata alone
• For Thermo .raw files, ThermoRawFileParser must be installed
• Detected parameters are empirical — they reflect what's IN the raw data, not what was intended
• If techsdrf detects a PTM not declared in the SDRF, confirm with the user before adding it (could be a biological signal or an artifact)
• If techsdrf does NOT detect a declared PTM, it may be at low abundance — do not automatically remove without user confirmation
• Always verify instrument ontology terms via OLS after techsdrf detection:

  mcp OLS → searchClasses(query="Q Exactive HF-X", ontologyId="ms")