tools

name: tools description: Repository-wide tooling including init wrapper, pack/publish utilities, and all helper scripts. Links to tools-special and tools-templates for subfolder details.

Repository Tools Overview

Continuous improvement: see continuous_improvement.md in this folder for notes on tool behaviour, past feedback, and update procedures.

Use this skill when working with repository-wide tools, understanding tool architecture, or coordinating workflows across multiple tool categories.

Tool categories

The scripts/ directory contains all helper scripts and utilities:

Core scripts (scripts/)

1. init.py: Async wrapper for pytextgen with mtime/inode caching

   • Discovers changed .md files (excludes .git, .obsidian, tools)
   • Caches (mtime, inode, text) to skip unchanged files
   • Normalizes newlines to \n before pytextgen processing
   • Passes through pytextgen flags (-C, --no-code-cache, --init-flashcards)
   • Commands: uv run -m init generate, uv run -m init clear

2. convert_wiki.py: Wikipedia HTML → Markdown converter

   • Reads HTML from clipboard
   • Normalizes links (relative paths with %20 encoding)
   • Downloads media to archives/Wikimedia Commons/
   • Uses convert_wiki.filename_rename_map.jsonc for filename renames
   • Preserves Wikipedia attribution
   • Command: uv run -m convert_wiki

3. pack.py: PageRank-ordered zip bundling

   • Walks Markdown links to build dependency graph
   • Computes PageRank to prioritize important files
   • Creates zip bundle with metadata (ranks, omissions, link closure)
   • Configurable: damping factor, iterations, max files
   • Command: uv run -m pack -o pack.zip -n 25 --damping-factor 0.5 --page-rank-iterations 100 <paths>

4. publish.py: Private → public history mirroring via git filter-repo

   • Clones private/.git temporarily
   • Runs git filter-repo with property Private-commit filtering
   • Rewrites commit history to remove sensitive paths
   • Rebases with signing, adds remote to public .git
   • Command: uv run -m publish --paths-file <file> (with literal:<path> lines)

Subfolder tools

• scripts/special/: Academic LMS converters and course management (see tools-special skill)

  • Canvas/HKUST Zinc converters
  • Course catalog fetchers

• scripts/templates/: Note scaffolding and pytextgen templates (see tools-templates skill)

  • new_wiki_page.py
  • pytextgen generate *.md templates

When to use this skill

• Understanding tool relationships and dependencies
• Coordinating multi-tool workflows (e.g., wiki ingestion → generation → packaging)
• Troubleshooting tool interactions
• Planning new tool development or refactoring

Common workflows

End-to-end wiki ingestion

1. Scaffold note: uv run -m templates.new_wiki_page (tools-templates)
2. Ingest HTML: uv run -m convert_wiki (convert_wiki.py)
3. Flashcard generation is automatic; do not run uv run -m init generate. Build workflows will handle it.

Academic course organization

1. Convert LMS export: uv run -m scripts.special.convert_canvas_submission (tools-special)
2. Update index: Edit special/academia/<Institution>/index.md
3. Add pytextgen fences; regeneration is handled by the build system and should not be invoked manually.

Packaging and publishing

1. Regeneration of generated content is automatic and occurs as part of the build; manual invocation (uv run -m init generate) is not required.
2. Package bundle: uv run -m pack -o bundle.zip -n 50 <paths> (pack.py)
3. Publish filtered history: uv run -m publish --paths-file paths.txt (publish.py)

Archive management

1. Archive content: Use pyarchivist tool (pyarchivist skill)
2. Verify index: Check archives/*/index.md updated
3. Reference in notes: Add relative links with %20 encoding

Tool dependencies

Python dependency metadata

Use pyproject.toml as the canonical source of Python dependencies:

• [project].dependencies: shared runtime dependencies
• [dependency-groups].dev: developer and test tooling
• [dependency-groups].scripts: the full union of packages referenced by inline # /// script metadata, even when a package also appears in [project].dependencies

For inline # /// script metadata, keep keys alphabetized (dependencies, requires-python, timestamp) and set requires-python = ">=3.13.0".

Install/update dependencies with bun install or uv sync.

External tools

• git: Required for all workflows
• git-filter-repo: Required for publish.py
• Python >=3.13.0: Repository-wide minimum version

Agent‑internal scripts policy

The repository occasionally contains small helper scripts used internally by the AI agent (for example, .github/scripts/validate-skills.py). These are not meant to be installed as CI tools, exposed to users, or added to package.json or the primary runtime dependency groups.

• Avoid creating similar "agent‑only" utilities without explicit approval from the repository owner.
• If an automated validation step or helper script is genuinely needed by the project, propose a formal PR and discuss with the owner before adding it to CI or packaging; the default assumption is that such tools belong in the .github/scripts/ directory and are not executed in production workflows.
• self/stash/**: Parent-repo scratch scripts rather than a git submodule
• private/**: Private content submodule

Tool architecture

init.py caching strategy

1. On first run: Walk workspace, compute (mtime, inode) for all .md files
2. Cache text and metadata in memory
3. On subsequent runs: Skip files with unchanged (mtime, inode)
4. Normalize \n before passing to pytextgen
5. Use -C / --no-cached to rebuild cache

Cache location: In-memory (not persisted to disk)

pytextgen compile cache

1. Compiled Python modules cached in __pycache__/
2. Use --no-code-cache to bypass
3. Clear with rm -rf __pycache__/ if stale

Git filter-repo workflow

1. Clone private/.git to temporary directory
2. Run git filter-repo --path-rename based on --paths-file
3. Filter commits by Private-commit property
4. Rebase and sign commits
5. Add temporary remote to public .git
6. User must manually push

CLI stability

Critical: Core tools have established interfaces; preserve:

• Argument names and order
• Expected input formats (clipboard, files, stdin)
• Output formats (Markdown, YAML, CSV, ZIP)
• Error codes and messages

If changes are needed, ask user for permission first.

Best practices

Tool coordination

• Regenerate before packaging: Generated content is normally kept fresh by the build process; manual uv run -m init generate -C is rarely needed and agents should not perform it.
• Clean before publishing: Verify private/ content is properly filtered before publish.py
• Archive before ingestion: Use pyarchivist for media before manual note creation
• Template before conversion: Scaffold frontmatter before ingesting content

Error handling

• Check tool exit codes before proceeding
• Verify file existence before processing
• Validate YAML/HTML/CSV formats before parsing
• Use --dry-run or preview modes when available

Performance

• Use init.py caching to skip unchanged files
• Parallelize independent operations (e.g., multiple convert_wiki runs)
• Limit PageRank iterations in pack.py for large graphs
• Use --exclude-extension in pack.py to skip large assets

When to ask for help

• If tool behavior is unexpected, consult user or check tool documentation
• If editing submodules (pytextgen, pyarchivist), ask user for permission
• If new tool is needed, discuss requirements and architecture with user
• If tool fails mysteriously, check Python version and dependencies

Common issues

1. Cache staleness: Use -C / --no-cached if init.py skips changed files
2. Module import errors: Ensure pyproject.toml dependencies are synced via bun install or uv sync
3. Git submodule out of date: Run git submodule update --remote
4. Path encoding issues: Ensure %20 encoding for spaces in links
5. Clipboard access: Some tools require clipboard support (may fail in headless environments)

Editing guidelines for scripts/*/.py

When editing Python helper scripts in scripts/:

• Preserve CLI surfaces: Keep argument names, defaults, and help text stable; avoid breaking uv run -m init generate/clear, uv run -m pack, uv run -m publish, and other tool entrypoints
• Maintain async/anyio patterns: Preserve async patterns using AnyIO APIs such as anyio.Path and anyio.Semaphore, but prefer importing helpers from Asyncer (e.g. from asyncer import create_task_group, soonify, asyncify) for better typing and editor support. The init wrapper has been refactored accordingly and includes its own thin _gather.
• Keep submodules read-only unless requested: private/** is a git submodule; ask user before editing.
• Normalize newlines: When touching the init wrapper, normalize to \n; do not bypass its exclusion list (.git, .obsidian, scripts)
• Favor relative imports: Use relative imports within the tools package; do not hardcode absolute host paths
• Keep inline script metadata synchronized: When editing a standalone script with inline # /// script metadata: (1) ensure the file begins with #!/usr/bin/env python shebang on line 1, (2) update the metadata together with [dependency-groups].scripts, (3) keep metadata keys alphabetized, and (4) retain requires-python = ">=3.13.0".
• Test thoroughly: Python tools are critical infrastructure; test changes carefully before committing