dataflows-authoring-cli

name: dataflows-authoring-cli description: > Create, update, delete, and manage Fabric Dataflows Gen2 artifacts with Power Query M mashup definitions via CLI (az rest / curl). Uses az rest and curl against the Fabric REST API to author definitions containing base64-encoded mashup.pq, queryMetadata.json, and .platform parts. Supports creating dataflows with inline definitions, modifying mashup queries, binding connections, triggering Execute refresh jobs with typed parameter overrides, and exporting definitions for CI/CD. Use when the user wants to: (1) create a new Dataflow Gen2 with Power Query M queries, (2) update a dataflow mashup definition, (3) trigger a dataflow refresh job, (4) bind or manage dataflow connections, (5) set up CI/CD via definition export and import, (6) delete a dataflow, (7) configure staging destinations. Triggers: "create dataflow", "author dataflow", "Power Query M", "mashup document", "update dataflow definition", "refresh dataflow", "dataflow connection", "ETL dataflow", "dataflow CI/CD".

Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.

• GitHub Copilot CLI / VS Code: invoke the check-updates skill.
• Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
• Skip if the check was already performed earlier in this session.

CRITICAL NOTES

1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

Dataflows Gen2 — Authoring via CLI

Table of Contents

Tool Stack

Agent check — verify before first operation:

az --version 2>/dev/null || echo "INSTALL: https://aka.ms/install-azure-cli"
jq --version 2>/dev/null || echo "INSTALL: apt-get install jq OR brew install jq"

Connection

Discover Workspace and Dataflow IDs

Per COMMON-CLI.md Finding Workspaces and Items in Fabric:

# List workspaces — find workspace ID by name
az rest --method get \
  --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces" \
  --query "value[?displayName=='MyWorkspace'].id" --output tsv

# List dataflows in workspace — find dataflow ID by name
WS_ID="<workspaceId>"
az rest --method get \
  --resource "https://api.fabric.microsoft.com" \
  --url "https://api.fabric.microsoft.com/v1/workspaces/$WS_ID/dataflows" \
  --query "value[?displayName=='MyDataflow'].id" --output tsv

Reusable Connection Variables

WS_ID="<workspaceId>"
DF_ID="<dataflowId>"
API="https://api.fabric.microsoft.com/v1"
RESOURCE="https://api.fabric.microsoft.com"

Agentic Workflows

Discover → Formulate → Execute → Verify

1. Discover → List workspaces, list dataflows, get definition (decode to inspect M code and metadata).
2. Formulate → Create or modify M code (mashup.pq), update queriesMetadata, prepare all 3 definition parts.
3. Execute → Create or update dataflow via REST API, optionally trigger refresh.
4. Verify → Poll LRO for completion, get definition again to confirm changes, check refresh status.

# 1. Discover — get current definition
RESULT=$(az rest --method post \
  --resource "$RESOURCE" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/getDefinition")

# Decode mashup.pq to inspect current queries
echo "$RESULT" | jq -r '.definition.parts[] | select(.path=="mashup.pq") | .payload' | base64 -d

# 2. Formulate — edit M code, prepare new definition parts
# (modify the decoded mashup, re-encode, build JSON payload)

# 3. Execute — update definition
az rest --method post \
  --resource "$RESOURCE" \
  --url "$API/workspaces/$WS_ID/dataflows/$DF_ID/updateDefinition?updateMetadata=true" \
  --body @definition.json

# 4. Verify — trigger refresh and poll
LOCATION=$(az rest --method post \
  --resource "$RESOURCE" \
  --url "$API/workspaces/$WS_ID/items/$DF_ID/jobs/Execute/instances" \
  --headers "Content-Length=0" \
  --output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')

Gotchas, Rules, Troubleshooting

For full authoring gotchas: DATAFLOWS-AUTHORING-CORE.md Gotchas and Troubleshooting. For CLI-specific issues: COMMON-CLI.md Gotchas & Troubleshooting (CLI-Specific).

MUST DO

• az login first — all az rest calls use the active session. No session → 401.
• Always --resource "https://api.fabric.microsoft.com" — wrong audience = 401.
• Base64-encode all definition parts — REST API expects payloadType: "InlineBase64".
• Send all 3 parts in updateDefinition — it's a full replacement, not incremental.
• Handle LRO polling — getDefinition, updateDefinition (with def), and job execution return 202; poll via Location header.
• Verify workspace has capacity — GET /v1/workspaces/{id} and check capacityId.
• Set formatVersion to "202502" in queryMetadata.json.
• Set loadEnabled: true for queries that should write output on refresh.

AVOID

• Hardcoded GUIDs — discover workspace/dataflow IDs via REST API (Connection section).
• Skipping base64 encoding — payloads will be rejected or corrupted.
• Partial definition updates — sending 1 or 2 parts silently drops queries.
• Creating dataflows with unbound connections — queries referencing nonexistent connectionId will fail at refresh.
• Using GET for getDefinition — it's a POST endpoint; GET returns 405.
• Constructing operation URLs manually — always use the Location header from 202 responses.
• Duplicate displayName values — not enforced but causes confusion.

PREFER

• az rest over raw curl — handles token acquisition and refresh automatically.
• getDefinition before updateDefinition — read-modify-write prevents accidental data loss.
• Validate M code before creating dataflow — syntax errors won't surface until refresh.
• jq for JSON manipulation — build definition payloads programmatically.
• "Automatic" for parameter type in job execution — lets engine infer from definition.
• ?updateMetadata=true on updateDefinition — ensures .platform changes (display name) are applied.
• Env vars (WS_ID, DF_ID, API, RESOURCE) for script reuse.

TROUBLESHOOTING

Examples

Example 1: Create a Dataflow Gen2 with Inline Definition

# 1. Prepare mashup.pq
MASHUP='section Section1;
shared Output = let
    Source = #table(type table [Name = text, Value = number],
        {{"Alpha", 100}, {"Beta", 200}, {"Gamma", 300}})
in Source;'

# 2. Prepare queryMetadata.json
QUERY_META='{
  "formatVersion": "202502",
  "queriesMetadata": {
    "Output": { "queryId": "00000000-0000-0000-0000-000000000001",
                "queryName": "Output", "loadEnabled": true }
  }
}'

# 3. Prepare .platform
PLATFORM='{"$schema":"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json","metadata":{"type":"Dataflow","displayName":"MyDataflow"},"config":{"version":"2.0","logicalId":"00000000-0000-0000-0000-000000000001"}}'

# 4. Base64-encode all parts
MASHUP_B64=$(echo -n "$MASHUP" | base64 -w0)
META_B64=$(echo -n "$QUERY_META" | base64 -w0)
PLAT_B64=$(echo -n "$PLATFORM" | base64 -w0)

# 5. Create the dataflow
az rest --method post \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Type=application/json" \
  --body "{
    \"displayName\": \"MyDataflow\",
    \"type\": \"Dataflow\",
    \"definition\": {
      \"parts\": [
        {\"path\": \"mashup.pq\",          \"payload\": \"${MASHUP_B64}\", \"payloadType\": \"InlineBase64\"},
        {\"path\": \"queryMetadata.json\", \"payload\": \"${META_B64}\",   \"payloadType\": \"InlineBase64\"},
        {\"path\": \".platform\",          \"payload\": \"${PLAT_B64}\",   \"payloadType\": \"InlineBase64\"}
      ]
    }
  }"

Example 2: Trigger a Refresh Job

# Trigger refresh (returns 202 + Location header for polling)
LOCATION=$(az rest --method post \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/jobs/instances?jobType=Pipeline" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Length=0" \
  --output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')

# Poll until complete
while true; do
  STATUS=$(az rest --method get --url "$LOCATION" \
    --resource "https://api.fabric.microsoft.com" --query "status" -o tsv)
  echo "Status: $STATUS"
  [[ "$STATUS" == "Completed" || "$STATUS" == "Failed" ]] && break
  sleep 10
done

Example 3: Update a Dataflow Definition

# Get current definition (POST, not GET — returns 202)
LOCATION=$(az rest --method post \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/getDefinition" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Length=0" \
  --output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')

# Poll for the definition
DEF=$(az rest --method get --url "${LOCATION}" \
  --resource "https://api.fabric.microsoft.com")

# Decode, modify, re-encode, then updateDefinition with all 3 parts
az rest --method post \
  --url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/updateDefinition" \
  --resource "https://api.fabric.microsoft.com" \
  --headers "Content-Type=application/json" \
  --body "{\"definition\": {\"parts\": [ ... ]}}"