climpred-forecast-verification

name: climpred-forecast-verification description: "Verify weather and climate forecasts using climpred. Use when computing forecast skill metrics (RMSE, ACC, CRPS, etc.), comparing hindcasts to observations, bootstrapping significance, removing bias, or working with HindcastEnsemble/PerfectModelEnsemble objects. Triggers on: forecast verification, prediction skill, hindcast, climate prediction, skill score, predictability." license: MIT

climpred — Forecast Verification

climpred verifies weather and climate forecasts. Built on xarray, dask, and xskillscore.

Docs: https://climpred.readthedocs.io | Docs for LLMs: https://climpred.readthedocs.io/en/latest/llms.txt

Install

pip install climpred[complete]
# or
conda install -c conda-forge climpred

Core Concepts

• init: initialization time dimension (forecast reference time)
• lead: forecast lead time dimension (with units attribute: "years", "months", "days", etc.)
• member: ensemble member dimension
• time: observation/verification time dimension

Two Ensemble Classes

HindcastEnsemble (real-world verification)

import climpred
from climpred import HindcastEnsemble
from climpred.tutorial import load_dataset

# initialized must have dims (init, lead, [member])
init = load_dataset("CESM-DP-SST")  # dims: (init, lead, member)
obs = load_dataset("ERSST")  # dims: (time,)
uninit = load_dataset("CESM-LE")  # dims: (time, member)

hindcast = HindcastEnsemble(init).add_observations(obs).add_uninitialized(uninit)

# Verify
skill = hindcast.verify(
    metric="rmse",
    comparison="e2o",
    dim="init",
    alignment="same_inits",
    reference=["persistence", "climatology", "uninitialized"],
)

PerfectModelEnsemble (model-internal verification)

from climpred import PerfectModelEnsemble

init = load_dataset("MPI-PM-DP-1D")  # dims: (init, lead, member)
control = load_dataset("MPI-control-1D")  # dims: (time,)

pm = PerfectModelEnsemble(init).add_control(control)
skill = pm.verify(metric="acc", comparison="m2e", dim=["init", "member"])

Key Methods

verify() Parameters

All keyword-only:

• metric (required): See reference/metrics.md
• comparison (required): See reference/comparisons.md
• dim: Dimensions to reduce. None = all except lead. Use [] for per-init skill.
• alignment (HindcastEnsemble only): "maximize", "same_inits", or "same_verifs"
• reference: ["persistence", "climatology", "uninitialized"] or subset
• groupby: Group init before verification (e.g., "month", "season")
• **metric_kwargs: Passed to metric (e.g., category_edges for rps)

bootstrap() Parameters

Same as verify() plus:

• iterations: Number of bootstrap iterations (recommend ≥500)
• sig: Significance level in percent (default: 95)
• resample_dim: "member" (default) or "init"

Returns Dataset with results dim: ["verify skill", "p", "low_ci", "high_ci"]

Comparisons

HindcastEnsemble: "e2o" (ensemble mean vs obs), "m2o" (members vs obs)

PerfectModelEnsemble: "m2m", "m2e", "m2c", "e2c"

Use "e2o" for deterministic metrics, "m2o" for probabilistic metrics (CRPS, RPS).

Common Metrics Quick Reference

Full metric details: See reference/metrics.md

Alignment (HindcastEnsemble only)

• "maximize": Most degrees of freedom; different inits per lead
• "same_inits": Same initializations across all leads
• "same_verifs": Same verification dates across all leads

Bias Removal

hindcast_corrected = hindcast.remove_bias(
    alignment="maximize",
    how="additive_mean",  # or "modified_quantile", "DetrendedQuantileMapping", etc.
    train_test_split="unfair",  # or "fair" (leave-one-out cross-validation)
    cv="LOO",  # cross-validation strategy
)

Common Workflows

1. Deterministic skill with references

skill = hindcast.verify(
    metric="acc",
    comparison="e2o",
    dim="init",
    alignment="same_inits",
    reference=["persistence", "climatology"],
)
skill["SST"].plot(hue="skill")

2. Probabilistic skill (CRPS)

skill = hindcast.verify(
    metric="crps",
    comparison="m2o",
    dim=["init", "member"],
    alignment="same_verifs",
)

3. Bootstrap significance

bs = hindcast.bootstrap(
    metric="acc",
    comparison="e2o",
    dim="init",
    alignment="same_inits",
    iterations=500,
    reference=["persistence", "climatology"],
)
# p < 0.05 means initialized beats reference

4. Per-init skill (no dimension reduction)

skill = hindcast.verify(
    metric="rmse",
    comparison="e2o",
    dim=[],
    alignment="same_verifs",
)

5. Seasonal groupby

skill = hindcast.verify(
    metric="acc",
    comparison="e2o",
    dim="init",
    alignment="same_inits",
    groupby="month",
)

Data Requirements

Initialized forecast (init):

• Must have init (initialization time) and lead dimensions
• lead must have a units attribute: "years", "months", "days", etc.
• Optional: member dimension for ensemble

Observations (obs):

• Must have time dimension
• Variables must match initialized dataset

Lead units attribute — set before creating ensemble:

ds["lead"].attrs["units"] = "years"

Tutorial Datasets

from climpred.tutorial import load_dataset

# call load_dataset() without args to list all available datasets

# Decadal (HindcastEnsemble)
init = load_dataset("CESM-DP-SST")
obs = load_dataset("ERSST")
uninit = load_dataset("CESM-LE")

# Perfect model
init_pm = load_dataset("MPI-PM-DP-1D")
control = load_dataset("MPI-control-1D")

# Subseasonal
s2s_init = load_dataset("ECMWF_S2S_Germany")
s2s_obs = load_dataset("Observations_Germany")

Gotchas

1. dim must match comparison: Use dim="init" with comparison="e2o" (no member dim). Use dim=["init", "member"] with comparison="m2o".
2. Lead units: Always set ds["lead"].attrs["units"] before creating the ensemble.
3. Integer init coords: Auto-converted to annual cftime. Use cftime or datetime init coords for sub-annual data.
4. alignment is required for HindcastEnsemble.verify() (not for PerfectModelEnsemble).
5. reference="uninitialized" requires .add_uninitialized() first.

Advanced: reference files

• Setting up data & toy examples: See reference/setting_up_data.md
• Bias removal guide: See reference/bias_removal.md
• Full metrics catalog: See reference/metrics.md
• Comparisons detail: See reference/comparisons.md