rdetoolkit

name: rdetoolkit description: > Guide development of RDE (Research Data Express) structured programs using rdetoolkit — a Python framework by NIMS for research data registration workflows. Covers project scaffolding, dataset function implementation, processing mode selection (Invoice / ExcelInvoice / MultiDataTile / RDEFormat), template editing, schema & metadata validation via CLI, encoding-safe file I/O with rdetoolkit.fileops, and CSV-to-graph generation with rdetoolkit.graph. MUST be used whenever code imports rdetoolkit, calls workflows.run(), reads/writes JSON in research-data contexts, processes CSV for graphing, edits invoice.schema.json or metadata-def.json, or runs rdetoolkit validate or rdetoolkit init commands. Also activate when the user mentions RDE, structured processing, NIMS, materials data, research data registration, or any rdetoolkit module. license: MIT metadata: author: nims-mdpf version: "1.0" docs: https://nims-mdpf.github.io/rdetoolkit/ repository: https://github.com/nims-mdpf/rdetoolkit

RDEToolKit — Structured Program Development Guide

RDEToolKit is a Python framework by NIMS (National Institute for Materials Science) that automates research data registration into RDE. It handles directory scaffolding, file validation, metadata extraction, thumbnail generation, and graph creation — so you only write the domain-specific data transformation logic.

Docs: https://nims-mdpf.github.io/rdetoolkit/ Repo: https://github.com/nims-mdpf/rdetoolkit

Quick Start

1. Initialize a project

pip install rdetoolkit
rdetoolkit init          # or: python3 -m rdetoolkit init

This generates the standard layout:

container/
├── main.py
├── requirements.txt
├── modules/
└── data/
    ├── inputdata/          # Place experimental data here
    ├── invoice/
    │   └── invoice.json
    └── tasksupport/
        ├── invoice.schema.json
        └── metadata-def.json

2. Write a dataset function (recommended signature)

from rdetoolkit.models.rde2types import RdeDatasetPaths

def dataset(paths: RdeDatasetPaths) -> None:
    # Read input from  paths.inputdata
    # Write outputs to paths.struct
    ...

3. Wire the entry point

import rdetoolkit
from modules.my_module import dataset

rdetoolkit.workflows.run(custom_dataset_function=dataset)

4. Run locally

python3 main.py

Critical Rules — Always Follow These

Use rdetoolkit APIs, Do NOT Reinvent

Research data files often use legacy encodings (Shift_JIS, EUC-JP, CP932). Standard Python open() / json.load() will crash on these files. Always use rdetoolkit's encoding-aware functions.

File I/O (rdetoolkit.fileops)

# ✅ CORRECT — handles Shift_JIS, EUC-JP, CP932 transparently
from rdetoolkit.fileops import read_from_json_file, write_to_json_file

metadata = read_from_json_file(paths.meta / "metadata.json")
write_to_json_file(paths.struct / "output.json", result)

# ❌ WRONG — will raise UnicodeDecodeError on legacy-encoded files
import json
with open(paths.meta / "metadata.json") as f:
    metadata = json.load(f)

CSV-to-Graph (rdetoolkit.graph)

For simple XY-axis graphs from CSV data, use csv2graph before writing matplotlib code. It generates publication-ready plots in one call.

from rdetoolkit.graph import csv2graph

# Generates XY line graph from CSV and saves to output directory
csv2graph(csv_path, output_dir)

See references/preferred-apis.md for full options and examples.

Metadata Writing (rdetoolkit.models.metadata.Meta)

ALWAYS use the Meta class to write metadata.json. Do NOT write it manually with json.dump().

from rdetoolkit.rde2util import Meta

def save_metadata(metadata: dict[str, str], metadata_def_json_path, save_path):
    meta = Meta(metadata_def_json_path)
    meta.assign_vals(metadata)       # All values MUST be strings
    meta.writefile(str(save_path))

Error Handling (Result Type — REQUIRED)

All helper functions in structured processing MUST use the Result type for error handling. Do NOT wrap the entire dataset() function in a single try/except block.

from rdetoolkit.result import Result, Success, Failure

def parse_data(filepath: Path) -> Result[pd.DataFrame, str]:
    try:
        # ... parsing logic ...
        return Success(df)
    except Exception as e:
        return Failure(f"Failed to parse: {e}")

def dataset(paths: RdeDatasetPaths) -> None:
    result = parse_data(paths.inputdata / "data.csv")
    if result.is_failure():
        raise RuntimeError(result.error)
    df = result.unwrap()

# ❌ WRONG: Giant try/except hides all errors
def dataset(paths: RdeDatasetPaths) -> None:
    try:
        # ... 100 lines ...
    except Exception as e:
        print(f"Error: {e}")

Dataset Function Signature

# ✅ RECOMMENDED — single-argument style (v1.4+)
from rdetoolkit.models.rde2types import RdeDatasetPaths

def dataset(paths: RdeDatasetPaths) -> None:
    ...

# ⚠️ LEGACY — two-argument style (still works, but do not use for new code)
from rdetoolkit.models.rde2types import RdeInputDirPaths, RdeOutputResourcePath

def dataset(inputdata: RdeInputDirPaths, output: RdeOutputResourcePath) -> None:
    ...

Path Access

Use the RdeDatasetPaths attributes. Do NOT hardcode paths.

Processing Modes

Choose the mode that matches your data registration scenario. Set it in rdeconfig.yaml under system.extended_mode.

Mode selection flowchart

How many files per registration?
├── One file → Invoice mode (default)
└── Multiple files
    ├── Each file needs different metadata?
    │   ├── Yes → ExcelInvoice mode
    │   └── No (shared metadata) → MultiDataTile mode
    └── Data already in RDE format? → RDEFormat mode

Configuration example

# rdeconfig.yaml
system:
  extended_mode: 'MultiDataTile'   # or 'ExcelInvoice', 'RDEFormat'
  save_raw: true
  magic_variable: true
  save_thumbnail_image: true

See references/modes.md for detailed mode descriptions and examples.

CLI Workflow — Correct Order Matters

Template editing and validation MUST follow this sequence. Running them out of order causes confusing validation errors.

Step 1: Edit templates (in this order)

1. data/tasksupport/invoice.schema.json — Define the schema first
2. data/tasksupport/metadata-def.json — Configure metadata definitions
3. data/invoice/invoice.json — Fill in values conforming to the schema

Step 2: Validate (in this order)

# 1. Check schema syntax itself
rdetoolkit validate invoice-schema data/tasksupport/invoice.schema.json

# 2. Check invoice conforms to schema
rdetoolkit validate invoice data/invoice/invoice.json \
  --schema data/tasksupport/invoice.schema.json

# 3. Check metadata definition
rdetoolkit validate metadata-def data/tasksupport/metadata-def.json

# 4. Full project validation (all of the above at once)
rdetoolkit validate all

Step 3: Run structured processing

python3 main.py

See references/cli-workflow.md for all CLI commands and CI/CD integration.

Project Structure Reference

container/
├── main.py                          # Entry point: calls workflows.run()
├── requirements.txt                 # Additional Python dependencies
├── modules/
│   └── my_module.py                 # Your dataset() function lives here
├── rdeconfig.yaml                   # Optional: mode & behavior config
└── data/
    ├── inputdata/
    │   └── <your experimental data>
    ├── invoice/
    │   └── invoice.json             # Data registration metadata
    └── tasksupport/
        ├── invoice.schema.json      # JSON Schema for invoice validation
        └── metadata-def.json        # Metadata field definitions

Building Structured Processing Autonomously

When asked to create a new RDE structured processing program, follow this sequence:

1. Analyze the user's input data file format and identify extractable metadata
2. Create metadata-def.json — define fields with bilingual names (ja/en) and types
3. Create invoice.schema.json — define the registration form schema
4. Create invoice.json — fill values conforming to the schema
5. Implement dataset() function — parse data, save metadata via Meta class, create structured CSV, generate plots
6. Wire main.py — rdetoolkit.workflows.run(custom_dataset_function=dataset)
7. Validate — rdetoolkit validate all, then python3 main.py

Each helper function in the dataset module MUST return a Result type. Metadata MUST be saved via the Meta class (not manual JSON writes). File I/O MUST use rdetoolkit.fileops.

If the user specifies a directory structure or coding pattern, follow their instructions. Otherwise, use the default patterns described here.

See references/building-structured-processing.md for the complete pattern with full code examples, directory specifications, metadata-def.json format, Meta class usage, Result-type error handling, and a submission checklist.

Common Mistakes and Fixes

References

• Full documentation: https://nims-mdpf.github.io/rdetoolkit/
• API reference: https://nims-mdpf.github.io/rdetoolkit/en/api/
• Repository: https://github.com/nims-mdpf/rdetoolkit
• Contributing guide: https://github.com/nims-mdpf/rdetoolkit/blob/main/CONTRIBUTING.md
• 5-mode templates: https://github.com/nims-mdpf/RDE_rdetoolkit_5mode_templates

Reference files in this skill

• references/building-structured-processing.md — Complete guide for building structured processing from scratch (dataset function pattern, metadata writing, Result-type error handling, directory specs, checklist)
• references/preferred-apis.md — Detailed fileops and csv2graph usage patterns
• references/modes.md — Deep dive into each processing mode
• references/cli-workflow.md — Complete CLI reference and CI/CD integration
• references/config.md — Configuration file specification (rdeconfig.yaml, pyproject.toml)