databricks-zerobus-ingest

name: databricks-zerobus-ingest description: "Build Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python/Java/Go/TypeScript/Rust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic." compatibility: Requires databricks CLI (>= v1.0.0) metadata: version: "0.1.0"

Zerobus Ingest

Build clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API.

Status: GA (Generally Available since February 2026; billed under Lakeflow Jobs Serverless SKU)

Documentation:

• Zerobus Overview
• Zerobus Ingest SDK
• Zerobus Limits

What Is Zerobus Ingest?

Zerobus Ingest is a serverless connector that enables direct, record-by-record data ingestion into Delta tables via gRPC. It eliminates the need for message bus infrastructure (Kafka, Kinesis, Event Hub) for lakehouse-bound data. The service validates schemas, materializes data to target tables, and sends durability acknowledgments back to the client.

Core pattern: SDK init -> create stream -> ingest records -> handle ACKs -> flush -> close

Quick Decision: What Are You Building?

If not specified, default to python.

Common Libraries

These libraries are essential for ZeroBus data ingestion:

• databricks-sdk>=0.85.0: Databricks workspace client for authentication and metadata
• databricks-zerobus-ingest-sdk>=1.0.0: ZeroBus SDK for high-performance streaming ingestion
• grpcio-tools These are typically NOT pre-installed on Databricks. Install them using execute_code tool:
• code: "%pip install databricks-sdk>=VERSION databricks-zerobus-ingest-sdk>=VERSION"

Save the returned cluster_id and context_id for subsequent calls.

Smart Installation Approach

Check protobuf version first, then install compatible

grpcio-tools import google.protobuf runtime_version = google.protobuf.version print(f"Runtime protobuf version: {runtime_version}")

if runtime_version.startswith("5.26") or runtime_version.startswith("5.29"): %pip install grpcio-tools==1.62.0 else: %pip install grpcio-tools # Use latest for newer protobuf

versions

Prerequisites

You must never execute the skill without confirming the below objects are valid:

1. A Unity Catalog managed Delta table to ingest into
2. A service principal id and secret with MODIFY and SELECT on the target table
3. The Zerobus server endpoint for your workspace region
4. The Zerobus Ingest SDK installed for your target language

See references/1-setup-and-authentication.md for complete setup instructions.

Minimal Python Example (JSON)

import json
from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties

sdk = ZerobusSdk(server_endpoint, workspace_url)
options = StreamConfigurationOptions(record_type=RecordType.JSON)
table_props = TableProperties(table_name)

stream = sdk.create_stream(client_id, client_secret, table_props, options)
try:
    record = {"device_name": "sensor-1", "temp": 22, "humidity": 55}
    stream.ingest_record(json.dumps(record))
    stream.flush()
finally:
    stream.close()

Detailed guides

You must always follow all the steps in the Workflow

Workflow

0. Display the plan of your execution
1. Determine the type of client
2. Get schema Always use references/4-protobuf-schema.md
3. Write Python code to a local file following the instructions in the relevant guide (e.g., scripts/zerobus_ingest.py)
4. Upload to workspace: databricks workspace import-dir ./scripts /Workspace/Users/<user>/scripts
5. Execute on Databricks using a job or notebook
6. If execution fails: Edit the local file, re-upload, and re-execute

Important

• Never install local packages
• Serverless limitation: The Zerobus SDK cannot pip-install on serverless compute. Use classic compute clusters, or use the Zerobus REST API (Beta) for notebook-based ingestion without the SDK.
• Explicit table grants: Service principals need explicit MODIFY and SELECT grants on the target table. Schema-level inherited permissions may not be sufficient for the authorization_details OAuth flow.

Execution Workflow

Step 1: Upload code to workspace

databricks workspace import-dir ./scripts /Workspace/Users/<user>/scripts

Step 2: Create and run a job

databricks jobs create --json '{
  "name": "zerobus-ingest",
  "tasks": [{
    "task_key": "ingest",
    "spark_python_task": {
      "python_file": "/Workspace/Users/<user>/scripts/zerobus_ingest.py"
    },
    "new_cluster": {
      "spark_version": "16.1.x-scala2.12",
      "node_type_id": "i3.xlarge",
      "num_workers": 0
    }
  }]
}'

databricks jobs run-now JOB_ID

If execution fails:

1. Read the error from the job run output
2. Edit the local Python file to fix the issue
3. Re-upload: databricks workspace import-dir ./scripts /Workspace/Users/<user>/scripts
4. Re-run: databricks jobs run-now JOB_ID

Installing Libraries

Databricks provides Spark, pandas, numpy, and common data libraries by default. Only install a library if you get an import error.

Add to the job configuration:

"libraries": [
  {"pypi": {"package": "databricks-zerobus-ingest-sdk>=1.0.0"}}
]

Or use init scripts in the cluster configuration.

🚨 Critical Learning: Timestamp Format Fix

BREAKTHROUGH: ZeroBus requires timestamp fields as Unix integer timestamps, NOT string timestamps. The timestamp generation must use microseconds for Databricks.

Key Concepts

• gRPC + Protobuf: Zerobus uses gRPC as its transport protocol. Any application that can communicate via gRPC and construct Protobuf messages can produce to Zerobus.
• JSON or Protobuf serialization: JSON for quick starts; Protobuf for type safety, forward compatibility, and performance.
• At-least-once delivery: The connector provides at-least-once guarantees. Design consumers to handle duplicates.
• Durability ACKs: Each ingested record returns a RecordAcknowledgment. Use flush() to ensure all buffered records are durably written, or use wait_for_offset(offset) for offset-based tracking.
• No table management: Zerobus does not create or alter tables. You must pre-create your target table and manage schema evolution yourself.
• Single-AZ durability: The service runs in a single availability zone. Plan for potential zone outages.

Common Issues

Related Skills

• databricks-python-sdk - General SDK patterns and WorkspaceClient for table/schema management
• databricks-pipelines - Downstream pipeline processing of ingested data
• databricks-unity-catalog - Managing catalogs, schemas, and tables that Zerobus writes to
• databricks-synthetic-data-gen - Generate test data to feed into Zerobus producers
• databricks-core - CLI install, profile selection, authentication

Resources

• Zerobus Overview
• Zerobus Ingest SDK
• Zerobus Limits