Metrics

This page documents the metrics module.

---

ordboost.metrics

Evaluation metrics for discrete ordinal and continuous probabilistic forecasts.

baseline_distribution(y_train, n_samples, boundary_epsilon=0.0001)

Construct a no-covariate baseline distribution.

Builds the unconditional empirical CDF of y_train and broadcasts it identically across n_samples rows, representing the best achievable forecast in the complete absence of covariate information -- the standard reference forecast against which skill scores are computed.

Parameters:

Name	Type	Description	Default
y_train	ArrayLike of shape (n_train_samples,)	Training targets defining the unconditional empirical distribution. Should be the full, unfiltered training set, even when scoring a filtered evaluation subset, so the baseline continues to represent "no covariate information" rather than "no information restricted to a subgroup" (see crps_skill_score).	required
n_samples	int	Number of rows to broadcast the baseline CDF across, e.g. the number of samples in the evaluation set this baseline will be scored against.	required
boundary_epsilon	float	Offset used to place a grid point strictly below the observed minimum of y_train, forced to CDF=0.0. Required because ContinuousPredictiveDistribution validates that every row's first value equals 0.0 within tolerance, and the empirical CDF at the observed minimum itself is generally nonzero (typically 1/n_train_samples, not 0).	1e-4

Returns:

Type	Description
ContinuousPredictiveDistribution	A distribution with n_samples identical rows, each equal to the empirical CDF of y_train.

Raises:

Type	Description
ValueError	If y_train is empty, or n_samples is not a positive integer.

Source code in ordboost/metrics.py

def baseline_distribution(
    y_train: ArrayLike, n_samples: int, boundary_epsilon: float = 1e-4
) -> ContinuousPredictiveDistribution:
    """Construct a no-covariate baseline distribution.

    Builds the unconditional empirical CDF of `y_train` and broadcasts it
    identically across `n_samples` rows, representing the best achievable
    forecast in the complete absence of covariate information -- the
    standard reference forecast against which skill scores are computed.

    Parameters
    ----------
    y_train : ArrayLike of shape (n_train_samples,)
        Training targets defining the unconditional empirical distribution.
        Should be the full, unfiltered training set, even when scoring a
        filtered evaluation subset, so the baseline continues to represent
        "no covariate information" rather than "no information restricted
        to a subgroup" (see `crps_skill_score`).
    n_samples : int
        Number of rows to broadcast the baseline CDF across, e.g.
        the number of samples in the evaluation set this baseline will be
        scored against.
    boundary_epsilon : float, default=1e-4
        Offset used to place a grid point strictly below the observed
        minimum of `y_train`, forced to CDF=0.0. Required because
        `ContinuousPredictiveDistribution` validates that every row's
        first value equals 0.0 within tolerance, and the empirical CDF at
        the observed minimum itself is generally nonzero (typically
        `1/n_train_samples`, not 0).

    Returns
    -------
    ContinuousPredictiveDistribution
        A distribution with `n_samples` identical rows, each equal to the
        empirical CDF of `y_train`.

    Raises
    ------
    ValueError
        If `y_train` is empty, or `n_samples` is not a positive integer.

    """
    y_train_arr = np.asarray(y_train, dtype=float)
    if y_train_arr.size == 0:
        raise ValueError("'y_train' must not be empty.")
    if not isinstance(n_samples, (int, np.integer)) or n_samples <= 0:
        raise ValueError(f"'n_samples' must be a positive integer, got {n_samples}.")

    unique_y = np.sort(np.unique(y_train_arr))
    unique_cdf = np.array([np.mean(y_train_arr <= v) for v in unique_y])

    grid_y = np.concatenate([[unique_y[0] - boundary_epsilon], unique_y])
    grid_cdf_row = np.concatenate([[0.0], unique_cdf])

    grid_cdf = np.tile(grid_cdf_row, (n_samples, 1))

    return ContinuousPredictiveDistribution(grid_y=grid_y, grid_cdf=grid_cdf)

crps_score(y_true, y_dist, sample_weight=None)

Compute the Continuous Ranked Probability Score (CRPS).

For discrete distributions, evaluates the exact squared cumulative probability error across threshold classes (a finite sum, not an approximation). For continuous predictive distributions, evaluates the exact integrated squared distance between the predicted CDF F(y) and the empirical step function I(y_true <= y), via scores.probability.crps_cdf. This computes the exact integral for a piecewise-linear CDF -- including correctly splitting the grid segment containing y_true -- rather than approximating it via trapezoidal integration over grid points alone, which systematically under- or over-estimates depending on where y_true falls within a segment (worse for wider bins).

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True physical target values.	required
y_dist	PredictiveDistribution	Predicted probability distribution object (discrete or continuous).	required
sample_weight	ArrayLike of shape (n_samples,)	Sample weights for weighted mean computation.	None

Returns:

Type	Description
float	The average CRPS across all samples (lower is better).

Raises:

Type	Description
ValueError	If y_true shape or sample count mismatches y_dist, or (for discrete distributions) y_true contains a value not present in y_dist.classes.

Source code in ordboost/metrics.py

def crps_score(
    y_true: ArrayLike,
    y_dist: Union[ContinuousPredictiveDistribution, DiscretePredictiveDistribution],
    sample_weight: Union[ArrayLike, None] = None,
) -> float:
    """Compute the Continuous Ranked Probability Score (CRPS).

    For discrete distributions, evaluates the exact squared cumulative
    probability error across threshold classes (a finite sum, not an
    approximation). For continuous predictive distributions, evaluates
    the exact integrated squared distance between the predicted CDF
    `F(y)` and the empirical step function `I(y_true <= y)`, via
    `scores.probability.crps_cdf`. This computes the exact integral for
    a piecewise-linear CDF -- including correctly splitting the grid
    segment containing `y_true` -- rather than approximating it via
    trapezoidal integration over grid points alone, which systematically
    under- or over-estimates depending on where `y_true` falls within a
    segment (worse for wider bins).

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True physical target values.
    y_dist : PredictiveDistribution
        Predicted probability distribution object (discrete or continuous).
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights for weighted mean computation.

    Returns
    -------
    float
        The average CRPS across all samples (lower is better).

    Raises
    ------
    ValueError
        If `y_true` shape or sample count mismatches `y_dist`, or (for
        discrete distributions) `y_true` contains a value not present in
        `y_dist.classes`.

    """
    y_true_arr = np.asarray(y_true, dtype=float)
    if y_true_arr.ndim != 1:
        raise ValueError(
            f"Expected 'y_true' to be a 1D array, got shape {y_true_arr.shape}."
        )

    n_samples = len(y_true_arr)

    if isinstance(y_dist, ContinuousPredictiveDistribution):
        if n_samples != y_dist.grid_cdf.shape[0]:
            raise ValueError(
                f"Sample count mismatch: 'y_true' has {n_samples} samples, but "
                f"'y_dist' has {y_dist.grid_cdf.shape[0]} samples."
            )

        fcst_da = xr.DataArray(
            y_dist.grid_cdf,
            dims=["sample", "threshold"],
            coords={"threshold": y_dist.grid_y},
        )
        obs_da = xr.DataArray(y_true_arr, dims=["sample"])

        result = crps_cdf(
            fcst_da, obs_da, threshold_dim="threshold", preserve_dims=["sample"]
        )
        sample_crps = result.total.values

    else:
        # Handle Discrete PredictiveDistribution
        if n_samples != y_dist.pmf.shape[0]:
            raise ValueError(
                f"Sample count mismatch: 'y_true' has {n_samples} samples, but "
                f"'y_dist' has {y_dist.pmf.shape[0]} samples."
            )
        if not set(y_true_arr).issubset(set(y_dist.classes)):
            missing_classes = set(y_true_arr) - set(y_dist.classes)
            raise ValueError(
                f"y_true contains target values not present in y_dist.classes: "
                f"{missing_classes}"
            )
        true_indicator = (
            y_true_arr[:, np.newaxis] <= y_dist.classes[np.newaxis, :]
        ).astype(float)
        cdf_diff_sq = (y_dist.cdf - true_indicator) ** 2
        sample_crps = np.sum(cdf_diff_sq, axis=1)

    if sample_weight is not None:
        weights = np.asarray(sample_weight, dtype=float)
        if weights.shape != (n_samples,):
            raise ValueError(
                f"Expected 'sample_weight' shape ({n_samples},), got {weights.shape}."
            )
        return float(np.average(sample_crps, weights=weights))

    return float(np.mean(sample_crps))

crps_skill_score(y_true, dist_model, dist_baseline, sample_weight=None)

Compute the CRPS skill score relative to a reference forecast.

Defined as CRPSS = 1 - CRPS(model) / CRPS(baseline). A value of 0 indicates no improvement over the baseline, 1 indicates a perfect forecast, and negative values indicate performance worse than the baseline. dist_baseline is typically constructed via baseline_distribution, but may be any reference forecast.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True physical target values.	required
dist_model	ContinuousPredictiveDistribution	The model's predicted distribution.	required
dist_baseline	ContinuousPredictiveDistribution	The reference forecast distribution to compare against.	required
sample_weight	ArrayLike of shape (n_samples,)	Sample weights, applied identically to both CRPS computations.	None

Returns:

Type	Description
float	The CRPS skill score.

Raises:

Type	Description
ValueError	If y_true's sample count mismatches dist_model or dist_baseline (raised by crps_score).

Source code in ordboost/metrics.py

def crps_skill_score(
    y_true: ArrayLike,
    dist_model: ContinuousPredictiveDistribution,
    dist_baseline: ContinuousPredictiveDistribution,
    sample_weight: Union[ArrayLike, None] = None,
) -> float:
    """Compute the CRPS skill score relative to a reference forecast.

    Defined as ``CRPSS = 1 - CRPS(model) / CRPS(baseline)``. A value of 0
    indicates no improvement over the baseline, 1 indicates a perfect
    forecast, and negative values indicate performance worse than the
    baseline. `dist_baseline` is typically constructed via
    `baseline_distribution`, but may be any reference forecast.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True physical target values.
    dist_model : ContinuousPredictiveDistribution
        The model's predicted distribution.
    dist_baseline : ContinuousPredictiveDistribution
        The reference forecast distribution to compare against.
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights, applied identically to both CRPS computations.

    Returns
    -------
    float
        The CRPS skill score.

    Raises
    ------
    ValueError
        If `y_true`'s sample count mismatches `dist_model` or
        `dist_baseline` (raised by `crps_score`).

    """
    crps_model = crps_score(y_true, dist_model, sample_weight=sample_weight)
    crps_baseline = crps_score(y_true, dist_baseline, sample_weight=sample_weight)
    return 1.0 - (crps_model / crps_baseline)

interval_coverage_rate(y_true, dist, alpha=0.1, sample_weight=None, tolerance=0.0)

Compute empirical coverage rate for a central prediction interval.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True continuous target values.	required
dist	ContinuousPredictiveDistribution	Predicted continuous distributions.	required
alpha	float	Tail significance level (e.g., alpha=0.10 specifies a 90% interval).	0.10
sample_weight	ArrayLike of shape (n_samples,)	Sample weights for weighted coverage computation.	None
tolerance	float	Slack applied to both bounds before the containment check. Use the mapper's boundary_epsilon so an atom placed boundary_epsilon inside the support limit is not counted as a miss.	0.0

Returns:

Type	Description
float	Proportion of true observations lying within predicted interval bounds.

Raises:

Type	Description
ValueError	If alpha not within (0.0, 1.0); or sample_weights do not have the same shape as y_arr.

Source code in ordboost/metrics.py

def interval_coverage_rate(
    y_true: ArrayLike,
    dist: ContinuousPredictiveDistribution,
    alpha: float = 0.10,
    sample_weight: Union[ArrayLike, None] = None,
    tolerance: float = 0.0,
) -> float:
    """Compute empirical coverage rate for a central prediction interval.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True continuous target values.
    dist : ContinuousPredictiveDistribution
        Predicted continuous distributions.
    alpha : float, default=0.10
        Tail significance level (e.g., alpha=0.10 specifies a 90% interval).
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights for weighted coverage computation.
    tolerance : float, default=0.0
        Slack applied to both bounds before the containment check. Use the
        mapper's ``boundary_epsilon`` so an atom placed ``boundary_epsilon``
        inside the support limit is not counted as a miss.

    Returns
    -------
    float
        Proportion of true observations lying within predicted interval bounds.

    Raises
    ------
    ValueError
        If alpha not within (0.0, 1.0); or sample_weights do not have
        the same shape as y_arr.

    """
    if not 0.0 < alpha < 1.0:
        raise ValueError("Significance level 'alpha' must lie within (0.0, 1.0).")

    y_true_arr = np.asarray(y_true, dtype=float)
    lower, upper = dist.interval(alpha=alpha)

    covered = (
        (y_true_arr >= lower - tolerance) & (y_true_arr <= upper + tolerance)
    ).astype(float)

    if sample_weight is not None:
        weights = np.asarray(sample_weight, dtype=float)
        if weights.shape != y_true_arr.shape:
            raise ValueError(
                f"Expected 'sample_weight' shape {y_true_arr.shape}, "
                f"got {weights.shape}."
            )
        return float(np.average(covered, weights=weights))

    return float(np.mean(covered))

marginal_calibration_curve(y_true, dist, mapper)

Compute the difference between the empirical CDF and the average CDF.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True continuous target values.	required
dist	ContinuousPredictiveDistribution	Predicted continuous distributions.	required
mapper	BaseBinMapper	The (fitted or unfitted) mapper instance whose floor_atom and ceiling_atom flags determine which grid points are treated as atoms. Only these two boolean attributes are read; the mapper does not need to be fitted.	required

Returns:

Name	Type	Description
grid_y	np.ndarray of shape (n_points,)	Grid points at which CDF were evaluated.
calibration	np.ndarray of shape (n_points,)	Marginal calibration.

Source code in ordboost/metrics.py

def marginal_calibration_curve(
    y_true: ArrayLike, dist: ContinuousPredictiveDistribution, mapper: BaseBinMapper
) -> tuple[np.ndarray, np.ndarray]:
    """Compute the difference between the empirical CDF and the average CDF.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True continuous target values.
    dist : ContinuousPredictiveDistribution
        Predicted continuous distributions.
    mapper : BaseBinMapper
        The (fitted or unfitted) mapper instance whose `floor_atom` and
        `ceiling_atom` flags determine which grid points are treated as
        atoms. Only these two boolean attributes are read; the mapper
        does not need to be fitted.

    Returns
    -------
    grid_y : np.ndarray of shape (n_points,)
        Grid points at which CDF were evaluated.
    calibration : np.ndarray of shape (n_points,)
        Marginal calibration.

    """
    y_true_arr = np.array(y_true, dtype=float)
    grid_y = dist.grid_y

    mean_cdf = dist.grid_cdf.mean(axis=0)
    empirical_cdf = np.array([np.mean(y_true_arr < x) for x in grid_y])

    if getattr(mapper, "floor_atom", False):
        mean_cdf[1] = 0.0  # left-hand limit at the floor atom
    if getattr(mapper, "ceiling_atom", False):
        empirical_cdf[-1] = 1  # by convention

    return grid_y, empirical_cdf - mean_cdf

pinball_loss(y_true, y_pred_q, q, sample_weight=None)

Compute the pinball (quantile) loss for a specific quantile level q.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True physical target labels.	required
y_pred_q	ArrayLike of shape (n_samples,)	Predicted target values at quantile level q.	required
q	float	Target quantile level in the range (0.0, 1.0).	required
sample_weight	ArrayLike of shape (n_samples,)	Sample weights for weighted mean computation.	None

Returns:

Type	Description
float	The average pinball loss across samples.

Raises:

Type	Description
ValueError	If q lies outside (0.0, 1.0) or array shapes do not match.

Source code in ordboost/metrics.py

def pinball_loss(
    y_true: ArrayLike,
    y_pred_q: ArrayLike,
    q: float,
    sample_weight: Union[ArrayLike, None] = None,
) -> float:
    """Compute the pinball (quantile) loss for a specific quantile level q.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True physical target labels.
    y_pred_q : ArrayLike of shape (n_samples,)
        Predicted target values at quantile level `q`.
    q : float
        Target quantile level in the range (0.0, 1.0).
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights for weighted mean computation.

    Returns
    -------
    float
        The average pinball loss across samples.

    Raises
    ------
    ValueError
        If `q` lies outside (0.0, 1.0) or array shapes do not match.

    """
    if not 0.0 < q < 1.0:
        raise ValueError(
            f"Quantile level 'q' must be strictly between 0.0 and 1.0, got {q}."
        )

    y_true_arr = np.asarray(y_true, dtype=float)
    y_pred_arr = np.asarray(y_pred_q, dtype=float)

    if y_true_arr.shape != y_pred_arr.shape:
        raise ValueError(
            f"Shape mismatch: 'y_true' shape {y_true_arr.shape} does not match "
            f"'y_pred_q' shape {y_pred_arr.shape}."
        )

    errors = y_true_arr - y_pred_arr
    loss = np.maximum(q * errors, (q - 1.0) * errors)

    if sample_weight is not None:
        weights = np.asarray(sample_weight, dtype=float)
        if weights.shape != y_true_arr.shape:
            raise ValueError(
                f"Expected 'sample_weight' shape {y_true_arr.shape}, "
                f"got {weights.shape}."
            )
        return float(np.average(loss, weights=weights))

    return float(np.mean(loss))

pinball_loss_skill_score(y_true, dist_model, dist_baseline, q, sample_weight=None)

Compute the pinball loss skill score at quantile level q.

Defined as PLSS = 1 - pinball_loss(model) / pinball_loss(baseline), both evaluated at the same quantile level q and against the same y_true. Interpretation mirrors crps_skill_score: 0 indicates no improvement over the baseline, 1 indicates a perfect forecast at that quantile, negative values indicate worse-than-baseline performance.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True physical target values.	required
dist_model	ContinuousPredictiveDistribution	The model's predicted distribution.	required
dist_baseline	ContinuousPredictiveDistribution	The reference forecast distribution to compare against.	required
q	float	Quantile level in (0.0, 1.0) at which to evaluate both forecasts.	required
sample_weight	ArrayLike of shape (n_samples,)	Sample weights, applied identically to both pinball loss computations.	None

Returns:

Type	Description
float	The pinball loss skill score at quantile q.

Raises:

Type	Description
ValueError	If q lies outside (0.0, 1.0), or if shapes mismatch (raised by pinball_loss).

Source code in ordboost/metrics.py

def pinball_loss_skill_score(
    y_true: ArrayLike,
    dist_model: ContinuousPredictiveDistribution,
    dist_baseline: ContinuousPredictiveDistribution,
    q: float,
    sample_weight: Union[ArrayLike, None] = None,
) -> float:
    """Compute the pinball loss skill score at quantile level `q`.

    Defined as ``PLSS = 1 - pinball_loss(model) / pinball_loss(baseline)``,
    both evaluated at the same quantile level `q` and against the same
    `y_true`. Interpretation mirrors `crps_skill_score`: 0 indicates no
    improvement over the baseline, 1 indicates a perfect forecast at that
    quantile, negative values indicate worse-than-baseline performance.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True physical target values.
    dist_model : ContinuousPredictiveDistribution
        The model's predicted distribution.
    dist_baseline : ContinuousPredictiveDistribution
        The reference forecast distribution to compare against.
    q : float
        Quantile level in (0.0, 1.0) at which to evaluate both forecasts.
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights, applied identically to both pinball loss
        computations.

    Returns
    -------
    float
        The pinball loss skill score at quantile `q`.

    Raises
    ------
    ValueError
        If `q` lies outside (0.0, 1.0), or if shapes mismatch (raised by
        `pinball_loss`).

    """
    y_pred_model = dist_model.ppf(q)
    y_pred_baseline = dist_baseline.ppf(q)
    loss_model = pinball_loss(y_true, y_pred_model, q=q, sample_weight=sample_weight)
    loss_baseline = pinball_loss(
        y_true, y_pred_baseline, q=q, sample_weight=sample_weight
    )
    return 1.0 - (loss_model / loss_baseline)

pit_diagnostics(y_true, dist, mapper, precision=2, resolution=1.0)

Construct exact PIT diagnostics using only per-sample scalars (O(n_samples)), not the full CDF grid (O(n_samples * n_grid_points)).

Each observation y is treated as the half-open bin [y, y + resolution), so its PIT value is uniform on [F(y), F(y + resolution)], where the CDF values are evaluated directly via dist.cdf(), rather than handing the entire grid_cdf array to Pit's threshold-dimension mode. Atoms at the floor and ceiling are handled separately, since they are point masses and not bins. The data from the distribution is rounded to the given precision, to avoid memory issues for large datasets, unless precision is None.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True target values, quantized to multiples of resolution (e.g. integer days).	required
dist	ContinuousPredictiveDistribution	Predicted continuous distributions, with grid_y/grid_cdf constructed by mapper.	required
mapper	BaseBinMapper	The (fitted or unfitted) mapper instance whose floor_atom and ceiling_atom flags determine which grid points are treated as atoms. Only these two boolean attributes are read; the mapper does not need to be fitted.	required
precision	int or None	Number of decimal places to round the PIT bounds to before constructing PitFcstAtObs. Pass None to disable rounding entirely.	2
resolution	float	Width of the interval represented by each observed value, i.e. the quantization step of the target (1.0 for integer-valued outcomes).	1.0

Returns:

Type	Description
PitFcstAtObs	A fitted PitFcstAtObs object exposing .hist_values(n_bins), .plotting_points(), .plotting_points_parametric(), .expected_value(), .variance(), and .alpha_score() for calibration diagnostics and summary statistics.

Raises:

Type	Description
ValueError	If y_true's length does not match dist.grid_cdf's sample count, if precision is not None and is not strictly positive, or if resolution is not strictly positive.
TypeError	If precision is not None and is not an integer.

Source code in ordboost/metrics.py

def pit_diagnostics(
    y_true: ArrayLike,
    dist: ContinuousPredictiveDistribution,
    mapper: BaseBinMapper,
    precision: Union[int, None] = 2,
    resolution: float = 1.0,
) -> PitFcstAtObs:
    """Construct exact PIT diagnostics using only per-sample scalars
    (O(n_samples)), not the full CDF grid (O(n_samples * n_grid_points)).

    Each observation `y` is treated as the half-open bin ``[y, y + resolution)``,
    so its PIT value is uniform on ``[F(y), F(y + resolution)]``, where the
    CDF values are evaluated directly via `dist.cdf()`, rather than handing
    the entire grid_cdf array to `Pit`'s threshold-dimension mode. Atoms at
    the floor and ceiling are handled separately, since they are point masses
    and not bins. The data from the distribution is rounded to the given
    precision, to avoid memory issues for large datasets, unless precision is
    `None`.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True target values, quantized to multiples of `resolution`
        (e.g. integer days).
    dist : ContinuousPredictiveDistribution
        Predicted continuous distributions, with `grid_y`/`grid_cdf`
        constructed by `mapper`.
    mapper : BaseBinMapper
        The (fitted or unfitted) mapper instance whose `floor_atom` and
        `ceiling_atom` flags determine which grid points are treated as
        atoms. Only these two boolean attributes are read; the mapper
        does not need to be fitted.
    precision : int or None, default=2
        Number of decimal places to round the PIT bounds to before
        constructing `PitFcstAtObs`. Pass `None` to disable rounding
        entirely.
    resolution : float, default=1.0
        Width of the interval represented by each observed value, i.e. the
        quantization step of the target (1.0 for integer-valued outcomes).

    Returns
    -------
    scores.probability.PitFcstAtObs
        A fitted `PitFcstAtObs` object exposing `.hist_values(n_bins)`,
        `.plotting_points()`, `.plotting_points_parametric()`,
        `.expected_value()`, `.variance()`, and `.alpha_score()` for
        calibration diagnostics and summary statistics.

    Raises
    ------
    ValueError
        If `y_true`'s length does not match `dist.grid_cdf`'s sample
        count, if `precision` is not None and is not strictly positive,
        or if `resolution` is not strictly positive.
    TypeError
        If `precision` is not None and is not an integer.

    """
    if precision is not None:
        if isinstance(precision, bool) or not isinstance(precision, (int, np.integer)):
            raise TypeError(
                f"Expected 'precision' to be an int or None, got "
                f"{type(precision).__name__}."
            )
        if precision <= 0:
            raise ValueError(f"'precision' must be greater than 0, got {precision}.")
    if resolution <= 0.0:
        raise ValueError(f"'resolution' must be greater than 0, got {resolution}.")

    y_true_arr = np.asarray(y_true, dtype=float)
    n_samples = dist.grid_cdf.shape[0]
    if y_true_arr.shape != (n_samples,):
        raise ValueError(
            f"Expected 'y_true' of shape ({n_samples},) to match 'dist', "
            f"got shape {y_true_arr.shape}."
        )

    # Interval [F(y), F(y + resolution)] for an observation in the bin [y, y + r)
    cdf_at_y = dist.cdf(y_true_arr)
    fcst_at_obs_left = cdf_at_y.copy()
    fcst_at_obs = dist.cdf(y_true_arr + resolution)

    if getattr(mapper, "floor_atom", False):
        floor_value = dist.grid_y[1]
        at_floor = np.isclose(y_true_arr, floor_value)
        # The atom is a point mass, not a bin: interval is [0, mass at floor]
        fcst_at_obs_left[at_floor] = 0.0
        fcst_at_obs[at_floor] = cdf_at_y[at_floor]

    if getattr(mapper, "ceiling_atom", False):
        ceiling_value = dist.grid_y[-1]
        # per-sample value just before the ceiling atom
        just_below_ceiling = dist.grid_cdf[:, -2]
        # bin directly below the atom must stop before the atom's mass
        below_ceiling = np.isclose(y_true_arr + resolution, ceiling_value)
        fcst_at_obs[below_ceiling] = just_below_ceiling[below_ceiling]
        # the atom itself: interval is [mass below ceiling, 1]
        at_ceiling = np.isclose(y_true_arr, ceiling_value)
        fcst_at_obs_left[at_ceiling] = just_below_ceiling[at_ceiling]
        fcst_at_obs[at_ceiling] = 1.0

    fcst_at_obs_left = np.clip(fcst_at_obs_left, 0.0, 1.0)
    fcst_at_obs = np.clip(np.maximum(fcst_at_obs, fcst_at_obs_left), 0.0, 1.0)

    if precision is not None:
        fcst_at_obs = np.round(fcst_at_obs, precision)
        fcst_at_obs_left = np.round(fcst_at_obs_left, precision)

    fcst_da = xr.DataArray(fcst_at_obs, dims=["sample"])
    fcst_left_da = xr.DataArray(fcst_at_obs_left, dims=["sample"])

    return PitFcstAtObs(fcst_da, fcst_at_obs_left=fcst_left_da)

pit_ks_test(pit)

Compute a Kolmogorov-Smirnov test of PIT uniformity.

Convenience wrapper around scipy.stats.kstest, using pit's duplicate-free plotting points. For discontinuous-CDF calibration assessment the alpha score is prefered and computed directly from the Pit object.

Parameters:

Name	Type	Description	Default
pit	PitFcstAtObs | Pit	A fitted Pit object, e.g. from pit_diagnostics.	required

Returns:

Name	Type	Description
statistic	float	The KS test statistic.
p_value	float	The associated p-value.

Source code in ordboost/metrics.py

def pit_ks_test(pit: PitFcstAtObs | Pit) -> tuple[float, float]:
    """Compute a Kolmogorov-Smirnov test of PIT uniformity.

    Convenience wrapper around `scipy.stats.kstest`, using `pit`'s
    duplicate-free plotting points. For discontinuous-CDF calibration
    assessment the alpha score is prefered and computed directly
    from the Pit object.

    Parameters
    ----------
    pit : PitFcstAtObs | Pit
        A fitted `Pit` object, e.g. from `pit_diagnostics`.

    Returns
    -------
    statistic : float
        The KS test statistic.
    p_value : float
        The associated p-value.

    """
    points = pit.plotting_points_parametric()
    result = kstest(points["x_plotting_position"].values, "uniform")
    return float(result.statistic), float(result.pvalue)

sharpness(dist, alpha=0.1)

Compute mean interval width at significance level alpha.

Parameters:

Name	Type	Description	Default
dist	ContinuousPredictiveDistribution	Predicted continuous distributions.	required
alpha	float	Tail significance level in range (0.0, 1.0).	0.10

Returns:

Type	Description
float	Mean interval width.

Raises:

Type	Description
ValueError	If alpha not within (0.0, 1.0)

Source code in ordboost/metrics.py

def sharpness(dist: ContinuousPredictiveDistribution, alpha: float = 0.10) -> float:
    """Compute mean interval width at significance level alpha.


    Parameters
    ----------
    dist : ContinuousPredictiveDistribution
        Predicted continuous distributions.
    alpha : float, default=0.10
        Tail significance level in range (0.0, 1.0).

    Returns
    -------
    float
        Mean interval width.

    Raises
    ------
    ValueError
        If alpha not within (0.0, 1.0)

    """
    if not 0.0 < alpha < 1.0:
        raise ValueError("Significance level 'alpha' must lie within (0.0, 1.0).")

    lower, upper = dist.interval(alpha=alpha)
    return float(np.mean(upper - lower))

winkler_score(y_true, dist, alpha=0.1, sample_weight=None)

Compute mean Winkler score for prediction intervals at significance level alpha.

Penalizes interval width and asymmetrically penalizes targets that fall outside the predicted lower and upper bounds.

Parameters:

Name	Type	Description	Default
y_true	ArrayLike of shape (n_samples,)	True continuous target values.	required
dist	ContinuousPredictiveDistribution	Predicted continuous distributions.	required
alpha	float	Tail significance level in range (0.0, 1.0).	0.10
sample_weight	ArrayLike of shape (n_samples,)	Sample weights for weighted mean computation.	None

Returns:

Type	Description
float	Mean Winkler score across samples (lower is better).

Raises:

Type	Description
ValueError	If alpha not within (0.0, 1.0)

Source code in ordboost/metrics.py

def winkler_score(
    y_true: ArrayLike,
    dist: ContinuousPredictiveDistribution,
    alpha: float = 0.10,
    sample_weight: Union[ArrayLike, None] = None,
) -> float:
    """Compute mean Winkler score for prediction intervals at significance level alpha.

    Penalizes interval width and asymmetrically penalizes targets that fall
    outside the predicted lower and upper bounds.

    Parameters
    ----------
    y_true : ArrayLike of shape (n_samples,)
        True continuous target values.
    dist : ContinuousPredictiveDistribution
        Predicted continuous distributions.
    alpha : float, default=0.10
        Tail significance level in range (0.0, 1.0).
    sample_weight : ArrayLike of shape (n_samples,), optional
        Sample weights for weighted mean computation.

    Returns
    -------
    float
        Mean Winkler score across samples (lower is better).

    Raises
    ------
    ValueError
        If alpha not within (0.0, 1.0)

    """
    if not 0.0 < alpha < 1.0:
        raise ValueError("Significance level 'alpha' must lie within (0.0, 1.0).")

    y_true_arr = np.asarray(y_true, dtype=float)
    lower, upper = dist.interval(alpha=alpha)

    width = upper - lower
    under_penalty = (2.0 / alpha) * (lower - y_true_arr) * (y_true_arr < lower)
    over_penalty = (2.0 / alpha) * (y_true_arr - upper) * (y_true_arr > upper)

    sample_scores = width + under_penalty + over_penalty

    if sample_weight is not None:
        weights = np.asarray(sample_weight, dtype=float)
        if weights.shape != y_true_arr.shape:
            raise ValueError(
                f"Expected 'sample_weight' shape {y_true_arr.shape}, "
                f"got {weights.shape}."
            )
        return float(np.average(sample_scores, weights=weights))

    return float(np.mean(sample_scores))