add-afm-model

name: add-afm-model description: Add support for a new HuggingFace MLX model to AFM. Use when user wants to add, onboard, or check compatibility of a model — handles everything from "already supported" to implementing new architectures. user_invocable: true

Add AFM Model

Investigate and add support for a HuggingFace MLX model to the AFM server.

Usage

• /add-afm-model <model-id> — e.g., /add-afm-model mlx-community/Qwen3-8B-4bit
• /add-afm-model <url> — e.g., /add-afm-model https://huggingface.co/mlx-community/Qwen3-8B-4bit

Instructions

Step 1: Parse Input

Extract the HuggingFace model ID from the user's input:

• Full URL: https://huggingface.co/org/model → org/model
• Model ID: org/model → use as-is
• If ambiguous, ask the user.

Step 2: Fetch config.json

Fetch https://huggingface.co/<model-id>/resolve/main/config.json using WebFetch.

Not MLX check: If config.json has no quantization or quantization_config field, the model is not MLX-quantized. Inform the user:

"This model is not in MLX format. Look for an MLX-quantized version on huggingface.co/mlx-community, or quantize it yourself with mlx_lm.convert."

Stop here if not MLX.

Step 3: Extract model_type

Read the model_type field from config.json. This is the key that maps to a Swift model implementation.

Also note: architectures, num_experts/num_local_experts (MoE indicator), image_token_id/vision_config (VLM indicator).

Step 4: Check LLMTypeRegistry

Search Scripts/patches/LLMModelFactory.swift for the model_type string in the LLMTypeRegistry.shared dictionary (lines ~25-80).

If found → Already supported. Tell the user:

"This model is already supported! Run it with:

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache afm mlx -m <model-id> --port 9999
```"

Suggest running /test-macafm for validation if this is a new model variant.

Stop here if already registered.

Step 5: Check Existing Architectures

The model_type is NOT in the registry. Now determine if an existing Swift implementation can handle it.

1. List files in vendor/mlx-swift-lm/Libraries/MLXLLM/Models/
2. Search for the model's base architecture name (e.g., if model_type is foo_moe, check for Foo.swift or similar)
3. Read the model's config.json fields and compare against existing implementations — some architectures handle variants (e.g., DeepseekV3 handles kimi_k2, Qwen2 handles acereason)
4. Check if an existing model has a dense fallback (e.g., numExperts == 0 path)

If a compatible architecture exists → Registry-only fix. Proceed to Tier: Registry Addition.

Step 6: Check Python mlx-lm

Search for the model_type in the Python mlx-lm library:

• Fetch https://github.com/ml-explore/mlx-lm/tree/main/mlx_lm/models (or search GitHub)
• Look for a Python file matching the model_type

If Python implementation exists → Port to Swift. Proceed to Tier: Port from Python.

If no Python implementation → Implement from scratch. Proceed to Tier: New Architecture.

Tier: Registry Addition

The simplest case — architecture exists, just needs a type alias.

1. Read references/implementation-guide.md for the patch system details
2. Add the model_type to LLMTypeRegistry.shared in Scripts/patches/LLMModelFactory.swift, mapping to the correct Configuration/Model pair
3. If the model is a VLM (has vision_config/image_token_id), also add to Scripts/patches/VLMModelFactory.swift
4. If the model has a new tool call format, update Scripts/patches/ToolCallFormat.swift infer() method
5. Apply patches: ./Scripts/apply-mlx-patches.sh
6. Build: swift build (or /build-afm for full rebuild)
7. Verify: start server and test with a simple prompt

Tier: Port from Python

Port the Python mlx-lm implementation to Swift.

1. Read references/implementation-guide.md for the full implementation pattern
2. Read references/model-investigation.md for config.json field mapping
3. Fetch and study the Python implementation from mlx-lm
4. Find the closest existing Swift model to use as a template (check vendor/mlx-swift-lm/Libraries/MLXLLM/Models/)
5. Create the new Swift file in Scripts/patches/<ModelName>.swift
6. Implement: Configuration (Codable) → Attention → MLP → TransformerBlock → Model
7. Add to PATCH_FILES, TARGET_PATHS, NEW_FILES in Scripts/apply-mlx-patches.sh
8. Register in LLMTypeRegistry (and VLM if needed)
9. Add weight sanitization if needed (in the Configuration's sanitize())
10. Apply, build, verify

Tier: New Architecture

No existing implementation anywhere. Research and implement from scratch.

1. Read references/implementation-guide.md and references/model-investigation.md
2. Find architecture documentation: paper, blog post, or reference implementation
3. Study config.json thoroughly for all architecture-specific fields
4. Find the closest existing Swift model as a starting template
5. Follow the same implementation steps as "Port from Python" above
6. Pay special attention to: attention patterns, normalization layers, MoE routing, positional embeddings
7. Test extensively — new architectures often have subtle bugs

Key Reminders

• NEVER edit files in vendor/ directly — all changes go through Scripts/patches/
• Always check VLM registry too if the model has vision capabilities
• Use MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache to avoid re-downloading
• After adding a new model, suggest running /test-macafm for full validation