oem-data-sync

name: oem-data-sync description: Run, monitor, and diagnose the 38 seed/enrich scripts that populate Supabase with OEM vehicle data (models, products, accessories, colors, pricing, portals, brochures)

OEM Data Sync

Wraps the existing dashboard/scripts/seed-*.mjs and enrich-*.mjs scripts with optional import_runs tracking. Each script fetches data from an OEM's API/website and upserts it into Supabase.

Auto-sync: variant_colors and specs_json are now automatically synced for all OEMs during every product upsert via orchestrator.syncVariantColors() and orchestrator.buildSpecsJson(). Individual spec columns (engine_size, cylinders, etc.) are also populated from meta_json on every upsert. Dedicated color seed scripts remain useful for OEM-specific sources (swatch images, gallery URLs, hero renders) that go beyond what the orchestrator auto-sync provides.

Adding a new OEM? Use the dashboard onboarding wizard at /dashboard/onboarding for guided setup, or see docs/OEM_ONBOARDING.md for the manual process.

Quick Start

# Run a single operation with import_run tracking
node skills/oem-data-sync/scripts/sync-runner.mjs --oem kgm-au --op accessories

# Run all weekly operations (dry-run first)
node skills/oem-data-sync/scripts/sync-runner.mjs --schedule weekly --dry-run

# Run a script directly (no tracking)
cd dashboard/scripts && node seed-kgm-accessories.mjs

Runner CLI

node skills/oem-data-sync/scripts/sync-runner.mjs [options]

Prerequisites

• Node.js 18+ with dashboard/node_modules/ installed (pnpm install in dashboard/)
• Supabase service role key hardcoded in each script (no .env needed)
• Network access to OEM APIs (some require specific headers, see edge cases below)

Operations Registry

Products (4 scripts)

Notes:

• Nissan products: Apigee needs apiKey + clientKey + publicAccessToken. Pricing requires browser-session tokens (not automated).
• Mitsubishi/Suzuki/KGM: Fully automated, no auth issues.

Accessories (10 scripts)

Notes:

• VW accessories: Requires guest token from homepage cookie. Token expires — if script fails, refetch homepage for fresh accessToken.
• Nissan accessories: HTML scraping with two different page templates. Fragile if Nissan redesigns.
• Mazda accessories: Extracts from React hydration ReactDOM.hydrate(...) inline data. Breaks if they change rendering.
• Kia accessories: Parses JSON-LD <script type="application/ld+json"> blocks. Rio and Niro return no data.
• VW has no model joins (accessories are categorized by type, not vehicle model).

Colors (10 scripts)

Notes:

• VW OneHub: get-onehub-offers API returns ALL on-sale variants with full color renders (4 angles + swatch tile), MRDP driveaway, MRRP, brochure PDFs, key features, hero images, and banner data. Version parameter auto-increments (start at 547). Dealer code 30140 used as reference. 49 variants, 410 colors, 15 model families.
• Chery colors: Extracted from data-json HTML attributes on model pages containing car_image (hero) and colour_image (swatch). Omoda sub-brand pages redirect to Chery C5/E5. * suffix in color name = premium paint.
• Nissan colors: AEM endpoint at /content/nissan_prod/en_AU/index/vehicles/browse-range/{slug}/version-explorer/jcr:content/core.versionexplorerdata.json. Navara MY26 on separate Storyblok microsite (no images). Juke needs model code remapping (30113 → 30304).
• Suzuki colors: Uses paintColours field (plural with 's'). Color types: Solid, Premium Metallic, Two-Tone Metallic.
• GWM colors: Match DB records by hex code (DB stores hex-{6chars}, Storyblok has real names + UUIDs).
• GMSV colors: Dual-source architecture. Trucks/SUV (Silverado, Yukon) scraped from GMSV AU HTML pages — color chips in <img> with colorizer/ path, jelly renders matched by alt text or slug. Corvettes (Stingray, E-Ray, Z06) from Chevrolet US colorizer JSON API (/content/gm/api/services/colorizerContent) — returns GM RPO codes, chip images, and 30-frame flipbook renders (frame 001 = front 3/4 hero). Critical: GM AEM pages contain chip texture images (pattern \d+ch-\w+-?\d+x\d+) in same colorizer/ directory as jelly renders — must filter these out or hero images show metallic paint close-ups instead of vehicles. ZR2 has no per-color jellies; uses generic fallback hero from nav.
• Foton colors: Tunland — color dots on /ute/tunland/ have label, image, and style (background-color) attributes. 4 variants (V7 C 4x2, V7 C 4x4, V9 L 4x4, V9 S 4x4) each get 8 colors (1 standard + 7 premium at +$690). Aumark S — 24 truck variants at /trucks/series/aumark-s/ come in White only; hero images scraped from static <img> tags per variant. Brochure/spec sheet PDFs extracted per-variant and stored in cta_links. All synced dynamically via syncFoton() in all-oem-sync.ts on daily cron.

Enrichment (3 scripts)

Offers (4 scripts)

Notes:

• GWM offers: Matches existing DB offers to Storyblok AUModel variants by normalized name + price. Updates hero_image_r2_key, price_amount (retail), and abn_price_amount (ABN). ABN pricing differs for utes/Tanks/Ora ($1,000 cheaper), same for Haval SUVs.
• KGM offers: Combines Payload CMS model data (ABN discounts, starting prices) with hardcoded website offer data (factory bonus, run-out sales). Images from CMS media (payload.therefinerydesign.com/api/media/file/). Deletes + re-inserts all KGM offers on each run.
• Kia offers: Two-tier HTML scraping. Tier 1 scrapes main offers page for 15 model-level offers (driveaway, finance, value-add) with hero images. Tier 2 follows detail page links to extract grade/variant pricing (Model Range section). Deduplicates shared detail pages. Deletes + re-inserts all Kia offers on each run. 52 offers (15 main + 37 variant), $22,140–$78,490.
• Subaru offers: 20 offers from /special-offers page (Inchcape AU Web Platform). Offers are national (not state-filtered — dealer picker only routes enquiries). 3 categories: driveaway (3), special editions (6), accessories (10), servicing (1). Images on S3 CDN (production-cdn-subaru-image-handler.s3.ap-southeast-2.amazonaws.com/offers/{slug}/). No API for offers — Retailer API covers models/variants/specs only. Obfuscated CSS Module classes require S3 CDN anchor extraction (in src/extract/engine.ts).

Banners (3 scripts)

Notes:

• seed-banners-v2.mjs: Server-side HTML scraping for Ford, GWM, KGM, Isuzu, Mazda, VW, Kia, Nissan. Each OEM has a custom cheerio extractor. Handles slick carousels, background-image CSS, picture/source elements, and responsive images.
• seed-banners-browser.mjs: Browser-based extraction for Hyundai, Toyota (client-side rendered carousels). Requires Chrome MCP session.
• seed-suzuki-banners.mjs: Suzuki /home/ hero carousel (3 slides) and /latest-offers/ hero banner. Uses background-image CSS with desktop/mobile variants.
• Video scripts: seed-ford-video.mjs (Brightcove Playback API), seed-banner-videos.mjs (Suzuki direct mp4).
• Ford video: Brightcove account 4082198814001, player H1RIrS7kf, policy key required. Mp4 URLs use Fastly signed tokens that expire — must call Playback API for fresh URLs.
• Missing OEMs: Subaru (Inchcape platform, obfuscated CSS Modules — offers extracted via S3 CDN anchor), Mitsubishi (no hero), LDV (data via Gatsby page-data.json, no banner extraction yet).

Ford Colors (1 script — replaces old enrichment)

Notes:

• Ford colors: Uses GPAS reference data (OEM-variants-main/data/ford/ford-variant-data.json + ford-color-guids.json). Replaces old enrich-ford-heroes.mjs (AEM colorizer HTML scraping). 388 colors with 100% hero/swatch/gallery from www.gpas-cache.ford.com.
• GWM Storyblok: Needs Origin https://www.gwmanz.com and Referer headers. Cache version (cv) parameter may need updating if Storyblok content changes.

Toyota (no script — browser-only)

Toyota data (21 models, 149 products, 802 colors, 132 pricing rows) was seeded via direct browser-to-Supabase REST API using Chrome MCP tools. Cloudflare protection blocks curl — all APIs require browser session with __cf_bm cookie. No automated seed script exists.

URL Rewrite (1 script)

Notes:

• Rewrites hero_image_url, swatch_url, and gallery_urls to point through the /media/:oemId/:encodedUrl Cloudflare Worker proxy.
• The Worker (src/routes/media.ts) fetches from the OEM CDN with appropriate headers and returns via Cloudflare edge cache (30-day TTL).
• Original OEM URLs stored in source_hero_url, source_swatch_url, source_gallery_urls columns.
• Change detection: if URL doesn't start with the Worker proxy prefix, it was freshly written by a seed script and needs rewriting.
• CLI: --oem <id>, --dry-run, --force. No R2 credentials needed.
• Should run AFTER all color seed/enrich scripts in the weekly schedule.

Portals (1 script)

Notes:

• Fetches portal credentials from Monday.com board via GraphQL API (account 229224, user 574175).
• Fuzzy-matches board item names to 19 OEM IDs.
• Extracts: portal_url, username, password, marketing_contact, guidelines_pdf_url (from Monday files column).
• 31 portals across all 19 OEMs. Platforms: sesimi, okta, dokio, sharepoint, box, fordimagelibrary, bms.hmc.co.kr, ateco.

Portal Assets (1 script)

Notes:

• Paginates the six FIL root categories (38441, 38443, 38444, 38445, 44149, 44150) via the allPublications search endpoint with fetchSize=1000. Recursive into subcategories.
• Upserts ~15k Ford assets keyed by (oem_id, external_source='ford_image_library', external_id=assetID).
• CDN URLs come from assetS3PreviewPath (JSON-encoded preview S3 URL) falling back to assetThumbnailPath. Both are public s3-ap-southeast-2.amazonaws.com/fil-filestor-storage-au URLs — no proxy needed.
• Asset-type classification derives from the filename extension (preferred) or fileFormat string. Critically, FIL's fileFormat values start with "Portable Network Graphics" / "Portable Document Format" — keyword matching must handle "portable" specifically or ~12k image rows get mis-classified as OTHER.
• Model parser covers ranger/everest/mustang/transit-custom/escape/puma/f-150/tourneo etc. Unparsed rate ~4%.
• Auth: JWT Bearer token scraped from a logged-in browser session (FORD_FIL_TOKEN env var). ~24h TTL. No automated refresh wired yet — the BrandHub login endpoint needs reverse-engineering (guessed POSTs to /UserService/v1/login etc. all return 401). Until then this operation is manual-trigger only.
• Sesimi/Algolia (Kia) uses a different pattern and is populated via dashboard/scripts/seed-kia-sesimi-assets.mjs — not part of this manifest yet.

Brochures (1 script)

Notes:

• Multi-OEM brochure URL scraper with per-OEM extractors. Updates vehicle_models.brochure_url.
• Kia: selectVehicleList API → brochure field (21/22 models).
• Toyota: Known Sitecore media library URLs — Cloudflare blocks HEAD checks, uses known URLs directly (15/21).
• Mazda: Central /brochures/ page with cheerio parsing (10/10).
• Ford: AEM DAM at /vehicles/download-brochure.html (8/13).
• Nissan: /vehicles/download-brochure.html with Nissan CDN (7/11).
• Mitsubishi: /buying-tools/download-brochure.html (5/6).
• GWM: Storyblok assets CDN (4/8).
• Isuzu: B-CDN models.json with brochure_pdfs field (2/2).
• Subaru: Predictable docs.subaru.com.au/{Model}-Brochure.pdf (2/7).
• Missing: Hyundai (0/17), Suzuki (0/7, form-gated), KGM (0/8), LDV (0/13).
• Total: 74/~179 models (46% coverage).

API Discovery (7 scripts)

Diagnosing Failures

Common Failure Modes

1. API Down / 5xx

• Symptom: Fetch failed: 500 or ECONNREFUSED
• Action: Wait and retry. Check if OEM site is down. Most recover within hours.

2. Auth Expired

• Affected: VW (guest token), Nissan (Apigee keys), Subaru (x-api-key)
• Symptom: 401 Unauthorized or 403 Forbidden
• VW fix: The script auto-fetches a fresh token from the homepage. If that fails, the homepage may have changed — inspect https://www.volkswagen-genuine-accessories.com/au/ for cookie changes.
• Subaru fix: API key w68ewXf97mnPODxXG6ZA4FKhP65wpUK576oLryW9 — check if still valid by hitting /api/v1/models.
• Nissan fix: Apigee apiKey/clientKey are embedded in the Nissan AU frontend bundle. Inspect network requests on nissan.com.au.

3. HTML Template Changed

• Affected: Mazda (React hydration), Kia (JSON-LD), Nissan (accessories HTML), Ford (AEM colorizer)
• Symptom: Script completes but inserts 0 rows, or crashes with parse error
• Action: Fetch the source page manually, check if data structure changed. Look for the extraction pattern:
  • Mazda: ReactDOM.hydrate(React.createElement(Koala.App, {props})) → extract "accessories":[
  • Kia: <script type="application/ld+json"> blocks with Schema.org Product objects
  • Nissan: Two HTML page templates for accessories
  • Ford: HTML-entity-encoded JSON in data-colorizer or similar attributes

4. Supabase Constraint Violation

• Symptom: duplicate key value violates unique constraint
• Action: Scripts do clean-delete + re-insert or upsert. If this fails, check if another process is writing to the same table. The onConflict fields are:
  • accessories: (oem_id, external_key)
  • vehicle_models: (oem_id, slug)
  • variant_colors: (product_id, color_slug)

5. Timeout

• Symptom: Script killed after timeout (default 120s)
• Action: Increase timeout with --timeout 300000. Some scripts (VW, Nissan) make many sequential API calls and can be slow.

6. Empty Response

• Symptom: Fetched 0 items or empty array
• Action: The OEM may have removed the endpoint or changed auth. Fetch the URL manually to verify.

Known Fragile Points

Database Tables Affected

import_runs Tracking

The sync-runner creates a record in import_runs for each script execution:

Schedules

Manifest

The machine-readable registry is at skills/oem-data-sync/sync-manifest.json. The runner reads it to determine which scripts to execute for a given --oem, --op, or --schedule filter.