data-sync

name: data-sync description: Sync and archive data from messaging platforms (WhatsApp, Discord, Slack, Twitter/X, Google) into Moltis memory as daily digest summaries. Orchestrates crawl tools and writes structured markdown to the memory system. origin: source: moltis url: https://github.com/moltis-org/moltis

data-sync — Crawl-to-Memory Pipeline

Orchestrate data archiving tools to sync external platform data into Moltis memory as searchable daily digest summaries.

Supported Sources

Workflow

For each source the user wants to sync:

1. Check prerequisites: Run <tool> doctor to verify the tool is installed and authenticated
2. Sync latest data: Run the tool's sync command to pull fresh data
3. Search/export with --json: Extract relevant messages and data
4. Summarize into daily digest: Create a structured markdown summary
5. Write to memory: Save as memory/<source>/YYYY-MM-DD.md

Memory Directory Convention

memory/
  whatsapp/YYYY-MM-DD.md    # wacrawl digests
  discord/YYYY-MM-DD.md     # discrawl digests
  slack/YYYY-MM-DD.md       # slacrawl digests
  twitter/YYYY-MM-DD.md     # birdclaw digests
  google/YYYY-MM-DD.md      # gog digests

Daily Digest Format

Each digest file should follow this structure for optimal memory indexing (section headers create natural chunk boundaries for search):

# <Source> — YYYY-MM-DD

## <Channel/Category 1>
- Key point or conversation summary
- Decisions made, action items
- Notable links or attachments shared

## <Channel/Category 2>
- ...

## Action Items
- [ ] Follow up on X with @person
- [ ] Review document shared by Y

Sync Commands by Source

WhatsApp (macOS only)

wacrawl sync
wacrawl search "query" --json --limit 50

Discord

discrawl sync --all
discrawl search "query" --json --after YYYY-MM-DD

Slack

slacrawl sync
slacrawl search "query" --json --after YYYY-MM-DD

Twitter/X

birdclaw sync
birdclaw inbox --json
birdclaw search "query" --json --after YYYY-MM-DD

Google

gog gmail list --query "after:YYYY/MM/DD" --json --limit 50
gog calendar events --from YYYY-MM-DD --to YYYY-MM-DD --json

Full Sync Example

When the user asks to "sync all my messages" or "update my data":

1. Check which tools are installed: which wacrawl discrawl slacrawl birdclaw gog
2. For each available tool, run doctor to check auth
3. Sync each authenticated source
4. Search for today's data with --json
5. Write daily digest summaries to memory

Guidelines

• Append, never overwrite: Daily digest files are append-only. If syncing multiple times per day, append new sections.
• Keep digests concise: Summarize conversations, don't dump raw messages. Aim for key decisions, action items, and notable information.
• Date in filename: Use YYYY-MM-DD.md format. The memory system indexes by filename date.
• 50 KB limit: Each memory write is limited to 50 KB. For high-volume days, focus on the most important items.
• No secrets in memory: Never write API tokens, passwords, or private keys into memory files.
• Ask before first sync: On first use, confirm with the user which sources they want to sync and how much history to pull.