external-data-upload

name: external-data-upload description: > Upload ITK test data to Filebase IPFS storage and produce .cid content links via the S3 REST API + npx ipfs-car (no Kubo daemon required), optionally mirror into ITKTestingData, and normalize existing .md5 / .sha256 / .cid content links. Use when the user wants to add test images, baseline data, or model files under Testing/Data/ or a module's data/ directory, or when asked to convert hash-based content links to CID. allowed-tools: - Bash - Read

ITK External Data Upload

Upload a file to Filebase IPFS storage and replace it with a .cid content link, maintain Testing/Data/content-links.manifest, and (optionally) mirror the bytes into ITKTestingData for the GitHub Pages gateway. Also: regenerate existing .md5 / .sha256 / .cid content links under the unixfs-v1-2025 profile.

Prerequisites

The developer must have the external-data-upload pixi environment installed and Filebase credentials exported. If not, direct them to README.md in this directory.

Required:

pixi environment installed: pixi install -e external-data-upload
Filebase IPFS bucket with an S3 access key
Environment variables exported: FILEBASE_ACCESS_KEY, FILEBASE_SECRET_KEY, FILEBASE_BUCKET

A local Kubo daemon, IPFS Desktop, or any ipfs pin remote PSA service is not required — the upload talks to Filebase's S3 REST API directly and relies on npx ipfs-car (installed via Node.js in the pixi environment) for local CAR construction.

Tasks this skill handles

1. Upload a single file

Run the upload script via pixi:

pixi run -e external-data-upload python \
    Utilities/Maintenance/ExternalDataUpload/upload.py <filepath>

If the user mentions ITKTestingData or asks you to mirror the bytes to GitHub Pages, pass --testing-data-repo <path>:

pixi run -e external-data-upload python \
    Utilities/Maintenance/ExternalDataUpload/upload.py \
    --testing-data-repo <path-to-ITKTestingData> \
    <filepath>

The script will:

Pack the file into a CARv1 with npx ipfs-car pack --no-wrap (defaults match the unixfs-v1-2025 profile)
Upload the CAR to the Filebase IPFS bucket via boto3 with Metadata={"import": "car"} and verify the CID returned by head_object matches the local CID
If --testing-data-repo given and file ≤ 50 MB, copy to <path>/CID/<cid> and git add it there. Files over 50 MB are skipped for the mirror step only (GitHub rejects > 50 MB) — Filebase pinning still succeeds.
Replace the source file with <source>.cid
Update Testing/Data/content-links.manifest

2. Normalize existing content links

Use when the user wants to convert .md5 / .sha256 / .sha512 links to .cid, or re-generate .cid links under the unixfs-v1-2025 profile.

pixi run -e external-data-upload python \
    Utilities/Maintenance/ExternalDataUpload/normalize.py <path-or-file>

Useful options:

--dry-run — report what would change
--hash-only — only touch .md5 / .shaNNN links, leave .cid alone
--cid-only — only re-hash existing .cid links under the new profile
--testing-data-repo <path> — forwarded to the upload helper
--bucket <name> — Filebase bucket override (default: $FILEBASE_BUCKET)

The normalize script fetches bytes through the gateway templates in CMake/ITKExternalData.cmake (same order as the build), verifies them against the declared hash or CID, and calls upload.upload_file_to_filebase to produce the new .cid.

3. Reconcile a local ITKTestingData mirror with Filebase and the manifest

Use when the user has files under <ITKTestingData>/CID/ that are referenced by .cid content links in the ITK source tree but are missing from Testing/Data/content-links.manifest (and so may not be pinned on Filebase yet).

pixi run -e external-data-upload python \
    Utilities/Maintenance/ExternalDataUpload/sync.py \
    --testing-data-repo <path-to-ITKTestingData>

The sync script:

Loads CIDs already in the manifest and skips files whose CID is listed.
Indexes .cid content links in the source tree via git ls-files.
For each file in <ITKTestingData>/CID/ whose CID is unlisted and is referenced by at least one .cid link, uploads the bytes to Filebase under the unixfs-v1-2025 profile and verifies the returned CID matches the filename.
Appends a manifest entry per referencing .cid link.

Files in <ITKTestingData>/CID/ not referenced by any .cid link are skipped silently — the script will not pin loose bytes. The source-tree .cid files are never modified.

Useful options:

--dry-run — preview the CIDs that would be uploaded
--bucket <name> — Filebase bucket override (default: $FILEBASE_BUCKET)

After Upload

Stage the git changes the upload script prints. Typical ITK workflow:

git rm <original-file>
git add <original-file>.cid
git add Testing/Data/content-links.manifest

If --testing-data-repo was used, follow the printed commands in that repo:

git -C <ITKTestingData> commit -m "Add <file> (<cid>)"
git -C <ITKTestingData> push

Commit the ITK changes with an appropriate prefix per Documentation/AI/git-commits.md:

ENH: for new test data
STYLE: for normalizing existing content links (no test semantics change)

How `.cid` Files Work

A .cid file is one line of plain text: a CIDv1, base32-encoded. ITK's CMake/ITKExternalData.cmake recognises the .cid extension and fetches through the gateway list declared there (local Kubo, ipfs.io, gateway.pinata.cloud, cloudflare-ipfs.com, dweb.link, plus the ITKTestingData GitHub Pages mirror at insightsoftwareconsortium.github.io/ITKTestingData/CID/<cid>).

Because CIDs are content-addressed, integrity is verified automatically at fetch time.