By name: write-kaggle-benchmarks description: Write, push, run, publish, and manage Kaggle Benchmark tasks using the kaggle CLI and the kaggle-benchmarks Python SDK. Use when the user wants to create or push a benchmark task (optionally with attached Kaggle datasets), run benchmarks against LLM models, check task/run status, stream or fetch execution logs, download results and source notebooks, publish a task to make it public, or troubleshoot benchmark workflows.
Write Kaggle Benchmarks
Keywords
Kaggle benchmarks, write a benchmark, benchmark task, kbench, push task, run task.
Official Resources
- SDK source & API — https://github.com/Kaggle/kaggle-benchmarks
- SDK auto-generated docs — https://deepwiki.com/Kaggle/kaggle-benchmarks
- CLI docs — https://github.com/Kaggle/kaggle-cli/blob/main/docs/benchmarks.md
Command Hierarchy
kaggle benchmarks (alias: kaggle b)
├── auth — Fetch Model Proxy credentials
├── init — Fetch credentials + setup local dev environment
└── tasks (alias: t) — Manage benchmark tasks
├── push — Upload a task from a .py file
├── run — Run a task against model(s)
├── list — List your benchmark tasks
├── status — Show task details and per-model run status
├── download — Download completed run outputs (and optionally source notebooks)
├── log (logs) — Show execution logs for run(s) (streams live for RUNNING runs)
├── publish — Make a task public (publishes the backing notebook by default)
├── models — List available benchmark models
└── delete — Delete a task (not yet supported by server)
Setup
# Full setup: credentials + .env + example_task.py + kaggle_benchmarks_reference.md
kaggle b init -y
# Credentials only (refresh MODEL_PROXY_* in .env)
kaggle b auth -y
Custom paths: --env-file <FILE> and --example-file <FILE> for init.
Env vars written by init:
MODEL_PROXY_URLMODEL_PROXY_API_KEYMODEL_PROXY_EXPIRY_TIMELLM_DEFAULTLLM_DEFAULT_EVALLLMS_AVAILABLE
Core workflow: Init → Write → Validate → Push → Run → Status → Download
Pacing — check in at every stage
Do NOT chain the full pipeline. Treat each numbered step below as a checkpoint:
- State what you are about to do for the current step (one sentence, including the exact command you intend to run).
- Wait for the user's go-ahead before executing — including for steps that look "obvious" like
initorlist. - After the step completes, show the relevant output, then stop. Do not auto-advance to the next step.
- Ask the user how they want to proceed: continue to the next documented step, change parameters, or branch off.
If the user explicitly asks for "the whole pipeline" or "do everything", you may chain, but summarize the planned chain in advance and ask for one confirmation covering the lot, instead of skipping the per-step checkpoints silently.
0. Init (once per environment, re-run when creds expire)
init fetches Model Proxy credentials, writes .env, and drops an
example_task.py + kaggle_benchmarks_reference.md next to it. Every
later step depends on the MODEL_PROXY_* vars it writes, so run it
before anything else — and re-run it any time python task.py or
kaggle b t run fails with an auth error (the API key is short-lived).
kaggle b init -y # first-time setup
kaggle b auth -y # creds-only refresh (no scaffolding)
1. Write a task file
A task file must:
- Import
kaggle_benchmarks as kbench - Define at least one function decorated with
@kbench.task(...) - Call
.run(kbench.llm)(or.evaluate(...)) on the task function — see Gotchas - Use
# %%cell markers (jupytext percent format)
Minimal example:
# %%
import kaggle_benchmarks as kbench
# %%
@kbench.task(name="my-test-task")
def my_test_task(llm):
response = llm.prompt("What is 2 + 2?")
kbench.assertions.assert_in("4", response, expectation="Should contain 4")
my_test_task.run(kbench.llm)
LLM resolution precedence (highest → lowest):
- Explicit model in code:
task.run(llm=kbench.llms["gemini-3.5-flash"]) - Default in code:
task.run(llm=kbench.llm)(resolves toLLM_DEFAULT) - Env vars from .env (
LLM_DEFAULT,LLMS_AVAILABLE,MODEL_PROXY_*)
2. Validate locally
Run the task end-to-end before pushing. This catches the silent-no-op gotcha and broken prompts before the push → run → wait → download round-trip.
kaggle b init -y # ensure .env is current
python task.py # run the task directly
ls -1 *.run.json # confirm a run file was produced
If python task.py exits cleanly and *.run.json appears, the task is safe to push. If validation fails, fix and re-run before proceeding to Step 3.
3. Push
kaggle b t push my-task -f task.py --wait
kaggle b t push my-task -f task.py -d owner/dataset1 -d owner/dataset2 # attach datasets
--wait [TIMEOUT] blocks until server-side creation finishes (no arg = indefinite). --poll-interval <SECONDS> caps the polling interval (default 60s; polling starts at 5s and grows adaptively). Repeat -d/--kaggle-dataset once per dataset (do not space-separate; see the Gotchas).
4. Run
# Interactive picker
kaggle b t run my-task
# Specific model
kaggle b t run my-task -m gemini-3.5-flash
# Multiple models (repeat -m, do NOT space-separate)
kaggle b t run my-task -m gemini-3.5-flash -m claude-haiku-4-5
# Wait for completion
kaggle b t run my-task -m gemini-3.5-flash --wait
List available models: kaggle b t models.
5. Status
kaggle b t status my-task
kaggle b t status my-task -m gemini-3.5-flash
Prints task metadata (slug, version, state, created timestamp, public flag, task URL) and a per-model run table. Errored runs render their final exception line under an Errors: section.
6. Download
kaggle b t download my-task # all terminal runs
kaggle b t download my-task -o ./results # custom directory
kaggle b t download my-task -m gemini-3.5-flash
kaggle b t download my-task -s # also fetch source notebooks
kaggle b t download my-task -f # force re-download (overwrite)
Output layout: <output>/<task>/<version>/<model>/<run_id>/.... Already-downloaded runs are skipped unless --force/-f is passed. With --include-source/-s, each run's directory also contains __notebook__.ipynb and __notebook_source__.ipynb alongside the regular outputs (useful for debugging the kernel session).
7. Log
kaggle b t log my-task # logs for every run of the task
kaggle b t log my-task -m gemini-3.5-flash # filter to one model
kaggle b t log my-task -m model-a -m model-b # multiple models, sequential
RUNNING runs stream live via SSE; COMPLETED/ERRORED runs print the persisted log in one shot; QUEUED runs print (No logs available — server returned 404) and continue.
8. Publish
kaggle b t publish my-task # publish task + backing notebook (default)
kaggle b t publish my-task --no-publish-backing-notebook # publish task only, keep notebook private
Publishes both the task and the backing notebook by default. If the task is already public the command is a no-op for the task itself but will still publish the notebook unless --no-publish-backing-notebook is passed.
Quick Recipes
Reminder: these are reference snippets, not invocations to chain automatically. Per the "Pacing" section above, run them one at a time with user confirmation between each, unless the user explicitly asks you to chain them.
# Push → run → download (run one command at a time, confirm between)
kaggle b t push my-task -f task.py --wait
kaggle b t run my-task -m gemini-3.5-flash --wait
kaggle b t download my-task -o ./results
# List tasks, filtered
kaggle b t list --name-regex "^math" --status errored
# Debug an errored run: pull logs first, then download source notebook
kaggle b t log my-task -m gemini-3.5-flash
kaggle b t download my-task -m gemini-3.5-flash -s -f
Gotchas
Most of these are silent failures the agent will not detect on its own — review before generating any task file or CLI invocation.
- No
.run()call → silent no-op. The push will succeed even if the file has no.run()(push validation only checks for@taskdecorators). The task will then execute on the server and produce no.run.json, so nothing is recorded. Every task function must end withtask_fn.run(kbench.llm)(or.evaluate(...)). MODEL_PROXY_API_KEYis short-lived. Ifpython task.pyfails with an auth error, re-runkaggle b auth -y(orkaggle b init -y) to refresh.init/authappend to the env file. Loaded viadotenvso last-wins makes re-running safe, but the file accumulates duplicate entries over time.- Task slug must match a
@taskdecorator.kaggle b t push <SLUG> -f file.pyfails if<SLUG>doesn't match the slugified name of some@kbench.task(name=...)(or function name) in the file. Names are normalized:My Task→my-task,my_task→my-task. - Use bare canonical model slugs (e.g.
gemini-3.5-flash,claude-haiku-4-5). The CLI auto-normalizes input by stripping any provider prefix (google/,anthropic/) and replacing@with-, sogoogle/gemini-3.5-flash,anthropic/claude-haiku-4-5@20251001, andclaude-haiku-4-5-6@defaultall work — but tables, error logs, and download directories always display the canonical form (e.g.claude-haiku-4-5-20251001). Standardize on the bare slug everywhere to keep commands, output, and paths consistent. - Re-pushing without
-ddetaches previous datasets. If a prior version of a task had Kaggle datasets attached, re-pushing without repeating the-dflags prints a yellow warning to stderr and silently detaches them. Always repeat the full-dlist on each push. -f -stogether to backfill source notebooks. If you originally downloaded without-s, a follow-updownload -swill skip cached runs and print a tip. Re-run with-f -sto overwrite cached runs and fetch their source notebooks.logis sequential, not interleaved. When multiple runs are active, the CLI blocks on the first run's stream until it terminates before moving to the next. This prevents log interleaving but means you'll wait on one run at a time.deleteis not implemented server-side. The command exists but currently printsDelete is not supported by the server yet.- Repeated flags, not space-separated. For multi-value flags (
-m,-d/--kaggle-dataset), pass the flag once per value:-m a -m b, not-m a b. Space-separated form is not supported and will error. - CLI scope is tasks only, not benchmarks. A benchmark is a curated collection of tasks. The CLI lets you create, push, and run individual tasks, but creating or managing benchmarks (collections) must be done on the Kaggle web UI.
SDK features beyond the basics
These are higher-leverage kaggle_benchmarks patterns from the task-writing skill. Reach for them when the minimal example isn't enough — and consult the upstream skill for full worked code.
Task definition rules
@kbench.benchmarkis an exact alias for@kbench.task. Use whichever reads better.- Return-type annotation is mandatory if the task returns a value.
def score(llm) -> float:— without-> float/-> bool/-> dictetc., the run object's.resulttyping breaks. store_task=Falsefor sub-tasks. If a@kbench.taskis called from inside another task (e.g. a per-row scorer in a dataset eval), setstore_task=Falseon the inner one to avoid persisting it as a top-level run.
Run object
After run = task_fn.run(...):
run.passed—bool,Trueif the result and every assertion passed.run.result— the returned value (typed by the task's return annotation).run.assertion_results—list[AssertionResult], every recorded assertion.
Reasoning and temperature on llm.prompt()
response = llm.prompt("Solve: 127 * 53?", reasoning="high") # "low" | "medium" | "high"
traces = kbench.last_reasoning_traces() # str | None — may be None if disabled/unsupported
response = llm.prompt("Write a creative story.", temperature=0.7) # default 0
Always null-check last_reasoning_traces() before using it.
Multimodal inputs
from kaggle_benchmarks.content_types import images, videos, audios
# Images: from_url / from_path / from_base64
img = images.from_url("https://example.com/cat.jpg") # llm.prompt auto-downloads + base64-encodes
img = images.from_path("local/chart.png") # read + base64 immediately
img = images.from_base64(b64_str, format="png") # default format="jpeg"
# Videos: YouTube URLs only (no local files, no non-YouTube URLs); select Gemini models only
video = videos.from_url("https://www.youtube.com/watch?v=...")
# Audio
audio = audios.from_path("speech.mp3")
response = llm.prompt("Describe this", image=img)
llm.prompt(...)auto-downloads image URLs to base64 (safe for non-URL-capable models).user.send(...)passes the raw URL through — use it to test native URL fetching, but it'll fail on models that don't support it.- Video input is YouTube-only and limited to specific Gemini models; local files or non-YouTube URLs will error.
Branching and isolating chat histories
chats.fork("name")— copy current history into a temporary branch; the original chat is untouched on exit.contexts.enter(chat=...)— low-level swap of the active chat for fully isolated agent histories. Use this only whenChatRoomdoesn't fit (e.g. agents that must not see each other).
Multi-agent: ChatRoom + Participant (Pattern I.5)
For debate / hidden-role / negotiation flows, prefer kbench.ChatRoom over hand-rolled contexts.enter() wiring. Each participant sees a perspective-projected transcript; the same kbench.llm can back many participants.
room = kbench.ChatRoom(system_prompt="A structured 2-turn debate.")
pro = room.add_participant(llm, name="Pro", system_prompt="Argue FOR.")
con = room.add_participant(llm, name="Con", system_prompt="Argue AGAINST.")
pro.reply()
con.reply()
See docs/chatroom/rooms_walkthrough.md in the kaggle-benchmarks repo for a full walkthrough.
Custom assertions
@assertion_handler()— decorate a function to register it as a custom assertion (returns/raises anAssertionResult). Use@assertion_handler(raises_assertion_error=True)to fail loudly on mismatch.kbench.assertions.assert_tool_was_invoked(fn)— verify the model actually called a registered tool during the prompt.