Metrics

Available metrics

Metrics are selected by name in the metrics setting of the configuration file. The tables list the names registered by default.

Classification

Name	Description	Uses
`accuracy`	Proportion of all classifications that were correct.	Class labels
`precision`	Proportion of positive classifications that are actually positive.	Class labels
`recall`	Proportion of actual positives that were classified correctly (true positive rate).	Class labels
`f1`	Harmonic mean of precision and recall.	Class labels
`roc_auc`, `auroc`	Area under the ROC curve. Both names are the same metric.	Decision function or probabilities
`ap`	Average precision, the area under the precision-recall curve.	Decision function or probabilities
`log_loss`	Logarithmic loss.	Probabilities
`brier_score`	Brier score loss.	Probabilities
`ici`	Integrated calibration index, the mean absolute difference between predicted probabilities and a spline-smoothed calibration curve.	Probabilities

Metrics that use class labels convert predicted probabilities to labels before they are computed.

Regression

Name	Description	Uses
`rmse`	Root mean squared error.	Point predictions
`mae`	Mean absolute error.	Point predictions
`crps`	Continuous ranked probability score, averaged over samples (lower is better).	Predictive distribution

Adding a metric

Describe the metric with a MetricSpec and register it with MetricRegistry. It can then be used by name in the configuration file.

from functools import partial

from sklearn.metrics import fbeta_score

from medpipe import MetricRegistry, MetricSpec

MetricRegistry.register_spec(
    MetricSpec(
        name="f2",
        func=partial(fbeta_score, beta=2),
        response_method="predict",
        display_name="F2",
        needs_threshold=True,
    )
)

Register the metric before creating the pipeline, so that the configuration is validated against it. A metric that uses a predictive distribution should define a per_sample_func, as explained in confidence intervals.

A complete script that registers a model, a preprocessing operation and a metric is shown in the examples.

core

Core metric functions module.

PredictionBundle `dataclass`

Pairs flat point predictions with an optional full predictive distribution, letting compute_metrics/bootstrap_confidence_intervals serve both point-based metrics (rmse, mae, accuracy, ...) and distribution-based metrics (crps) from a single call.

A bare numpy.ndarray remains accepted everywhere a PredictionBundle is accepted, and is treated as PredictionBundle(point=array, dist=None) - existing point-array-only callers are unaffected.

Attributes:

Name	Type	Description
`point`	`npt.NDArray or None, default=None`	Flat point predictions (or probabilities/decision scores for classification), used by every metric except `predict_dist`-based ones.
`dist`	`PredictiveDistribution or None, default=None`	A full predictive distribution object (e.g. returned by `DistributionalPipeline.predict_dist`), used only by `predict_dist`-based metrics such as crps. Duck-typed objects exposing scipy-style `.ppf(q)` and `.cdf(x)` (e.g. NGBoost's) are also accepted at runtime by the crps metric.

Source code in medpipe/metrics/core.py

@dataclass(frozen=True)
class PredictionBundle:
    """Pairs flat point predictions with an optional full predictive
    distribution, letting `compute_metrics`/`bootstrap_confidence_intervals`
    serve both point-based metrics (rmse, mae, accuracy, ...) and
    distribution-based metrics (crps) from a single call.

    A bare `numpy.ndarray` remains accepted everywhere a `PredictionBundle`
    is accepted, and is treated as `PredictionBundle(point=array, dist=None)`
    - existing point-array-only callers are unaffected.

    Attributes
    ----------
    point : npt.NDArray or None, default=None
        Flat point predictions (or probabilities/decision scores for
        classification), used by every metric except `predict_dist`-based
        ones.
    dist : PredictiveDistribution or None, default=None
        A full predictive distribution object (e.g. returned by
        `DistributionalPipeline.predict_dist`), used only by
        `predict_dist`-based metrics such as crps. Duck-typed objects exposing
        scipy-style `.ppf(q)` and `.cdf(x)` (e.g. NGBoost's) are also
        accepted at runtime by the crps metric.

    """

    point: npt.NDArray | None = None
    dist: PredictiveDistribution | None = None

ici_score

ici_score(
    y: NDArray, y_pred: NDArray, cv_splines: int = 3
) -> float

Computes the integrated calibration index using a spline-based curve.

Parameters:

Name	Type	Description	Default
`y`	`NDArray`	Ground truth binary target labels.	required
`y_pred`	`NDArray`	Predicted probabilities of shape (n_samples,) or (n_samples, 2).	required
`cv_splines`	`int`	Number of internal cross-validation folds `SplineCalib` uses to select its regularization strength. Lower values fit faster (this runs once per bootstrap resample when `ici` is evaluated with a confidence interval) at the cost of a coarser regularization search; `SplineCalib`'s own default is 5.	`3`

Returns:

Type	Description
`float`	The integrated calibration index, the mean absolute difference between the predicted probabilities and their spline-smoothed values (lower is better).

Raises:

Type	Description
`ValueError`	If the spline calibration fails to produce predictions.

Source code in medpipe/metrics/core.py

def ici_score(y: npt.NDArray, y_pred: npt.NDArray, cv_splines: int = 3) -> float:
    """Computes the integrated calibration index using a spline-based curve.

    Parameters
    ----------
    y : npt.NDArray
        Ground truth binary target labels.
    y_pred : npt.NDArray
        Predicted probabilities of shape (n_samples,) or (n_samples, 2).
    cv_splines : int, default=3
        Number of internal cross-validation folds `SplineCalib` uses to
        select its regularization strength. Lower values fit faster (this
        runs once per bootstrap resample when `ici` is evaluated with a
        confidence interval) at the cost of a coarser regularization
        search; `SplineCalib`'s own default is 5.

    Returns
    -------
    float
        The integrated calibration index, the mean absolute difference
        between the predicted probabilities and their spline-smoothed
        values (lower is better).

    Raises
    ------
    ValueError
        If the spline calibration fails to produce predictions.

    """
    if y_pred.ndim == 2:
        y_pred = y_pred[:, 1]

    spline = SplineCalib(cv_spline=cv_splines)
    spline.fit(y_pred, y)
    smoothed_outputs = spline.predict(y_pred)

    if smoothed_outputs is not None:
        return float(np.mean(np.abs(smoothed_outputs - y_pred)))
    else:
        raise ValueError("Error predicting probabilities with spline")

crps_score

crps_score(
    y: NDArray, dist: PredictiveDistribution
) -> float

Computes the mean Continuous Ranked Probability Score (CRPS) for a predictive distribution. See _crps_values for the per-sample computation.

Parameters:

Name	Type	Description	Default
`y`	`NDArray`	Ground truth continuous target values of shape (n_samples,).	required
`dist`	`PredictiveDistribution`	A predictive distribution object exposing `.ppf(q)` and `.cdf(x)`.	required

Returns:

Type	Description
`float`	The mean CRPS across samples (lower is better).

Source code in medpipe/metrics/core.py

def crps_score(y: npt.NDArray, dist: PredictiveDistribution) -> float:
    """Computes the mean Continuous Ranked Probability Score (CRPS) for a
    predictive distribution. See `_crps_values` for the per-sample
    computation.

    Parameters
    ----------
    y : npt.NDArray
        Ground truth continuous target values of shape (n_samples,).
    dist : PredictiveDistribution
        A predictive distribution object exposing `.ppf(q)` and `.cdf(x)`.

    Returns
    -------
    float
        The mean CRPS across samples (lower is better).

    """
    return float(np.mean(_crps_values(y, dist)))

build_scorers

build_scorers(
    metrics: list[str] | NDArray,
) -> dict[str, Callable]

Build the dictionary of scorers for cross-validation.

Parameters:

Name	Type	Description	Default
`metrics`	`list[str]`	List of metrics to use.	required

Returns:

Name	Type	Description
`scorers`	`dict[str, Callable]`	Dictionary of scorers to pass to cross_validate.

Raises:

Type	Description
`TypeError`	If metrics is not a list of strings.
`ValueError`	If a metric is not a valid option.

Source code in medpipe/metrics/core.py

def build_scorers(metrics: list[str] | npt.NDArray) -> dict[str, Callable]:
    """
    Build the dictionary of scorers for cross-validation.

    Parameters
    ----------
    metrics : list[str]
        List of metrics to use.

    Returns
    -------
    scorers : dict[str, Callable]
        Dictionary of scorers to pass to cross_validate.

    Raises
    ------
    TypeError
        If metrics is not a list of strings.
    ValueError
        If a metric is not a valid option.

    """
    if not isinstance(metrics, (list, np.ndarray)):
        raise TypeError("Input metrics should be a list of strings")
    if len(metrics) == 0 or not isinstance(metrics[0], str):
        raise TypeError("Input metrics should be a list of strings")

    scorers = {}
    for metric_name in metrics:
        spec = MetricRegistry.get(metric_name)
        scorers[metric_name] = spec.get_scorer()

    return scorers

compute_metrics

compute_metrics(
    metrics: list[str] | NDArray,
    y: NDArray,
    y_pred: NDArray | PredictionBundle,
) -> NDArray

Computes metrics based on predicted data.

Scores are located at the same index as in the metrics array.

Parameters:

Name	Type	Description	Default
`metrics`	`list[str]`	List of metrics to use.	required
`y`	`NDArray`	Ground truth labels.	required
`y_pred`	`NDArray or PredictionBundle`	Predictions from the model, either a flat array of shape (n_samples,) or (n_samples, 2), or a `PredictionBundle` carrying both flat point predictions and a full predictive distribution (required for `predict_dist`-based metrics such as crps).	required

Returns:

Name	Type	Description
`scores`	`NDArray`	Score array of shape (n_metrics,).

Raises:

Type	Description
`TypeError`	If metrics is not a list of strings. If y is not a np.ndarray. If y_pred is not a np.ndarray or PredictionBundle.
`ValueError`	If there is only one class for AUROC and AP calculations. If a `predict_dist`-based metric is requested but no distribution is available. If the distribution and y have different numbers of samples.

Source code in medpipe/metrics/core.py

def compute_metrics(
    metrics: list[str] | npt.NDArray,
    y: npt.NDArray,
    y_pred: npt.NDArray | PredictionBundle,
) -> npt.NDArray:
    """
    Computes metrics based on predicted data.

    Scores are located at the same index as in the metrics array.

    Parameters
    ----------
    metrics : list[str]
        List of metrics to use.
    y : npt.NDArray
        Ground truth labels.
    y_pred : npt.NDArray or PredictionBundle
        Predictions from the model, either a flat array of shape
        (n_samples,) or (n_samples, 2), or a `PredictionBundle` carrying
        both flat point predictions and a full predictive distribution
        (required for `predict_dist`-based metrics such as crps).

    Returns
    -------
    scores : npt.NDArray
        Score array of shape (n_metrics,).

    Raises
    ------
    TypeError
        If metrics is not a list of strings.
        If y is not a np.ndarray.
        If y_pred is not a np.ndarray or PredictionBundle.
    ValueError
        If there is only one class for AUROC and AP calculations.
        If a `predict_dist`-based metric is requested but no distribution
        is available.
        If the distribution and y have different numbers of samples.

    """
    if not isinstance(metrics, (list, np.ndarray)):
        raise TypeError("Input metrics should be a list of strings")
    if len(metrics) == 0 or not isinstance(metrics[0], str):
        raise TypeError("Input metrics should be a list of strings")

    if not isinstance(y_pred, (np.ndarray, PredictionBundle)):
        raise TypeError(f"Input y_pred should be a np.ndarray, but got {type(y_pred)}")

    if not isinstance(y, np.ndarray):
        raise TypeError(f"Input y should be a np.ndarray, but got {type(y)}")

    # Ensure y contains at least 2 classes for binary ranking/precision metrics
    if len(np.unique(y)) < 2:
        binary_metrics = {"roc_auc", "auroc", "ap"}
        if any(m in binary_metrics for m in metrics):
            raise ValueError(
                "Only one class present in y_true. "
                "AUROC and AP score are not defined in that case."
            )

    if isinstance(y_pred, PredictionBundle):
        bundle = y_pred
    else:
        point = y_pred[:, 1] if y_pred.ndim == 2 else y_pred
        bundle = PredictionBundle(point=point, dist=None)

    _check_dist_length(y, bundle.dist)

    scores = np.zeros(len(metrics))

    for i, metric_name in enumerate(metrics):
        spec = MetricRegistry.get(metric_name)

        if spec.response_method == "predict_dist":
            if bundle.dist is None:
                raise ValueError(
                    f"'{metric_name}' requires a full predictive distribution "
                    "and cannot be computed from a flat prediction array via "
                    "compute_metrics(); pass a PredictionBundle with `dist` set."
                )
            scores[i] = float(spec.func(y, bundle.dist))
        elif bundle.point is None:
            raise ValueError(
                f"'{metric_name}' requires point predictions, but the "
                "PredictionBundle has `point=None`."
            )
        elif spec.needs_threshold:
            scores[i] = float(spec.func(y, np.round(bundle.point)))
        else:
            scores[i] = float(spec.func(y, bundle.point))

    return scores

bootstrap_confidence_intervals

bootstrap_confidence_intervals(
    metrics: list[str],
    y_true: NDArray,
    y_pred: NDArray | PredictionBundle,
    n_bootstraps: int = 1000,
    ci_level: float = 0.95,
    random_state: int | Generator | None = None,
    n_jobs: int | None = 1,
) -> dict[str, dict[str, float]]

Compute non-parametric bootstrap confidence intervals for metrics.

Resamples paired target labels and predictions with replacement, evaluating all specified metrics across bootstrap iterations. Edge cases where resamples lack sufficient class diversity (e.g., single-class bootstrap draws) are safely caught and skipped.

Parameters:

Name	Type	Description	Default
`metrics`	`list[str]`	Metric identifier keys registered in `MetricRegistry` (e.g., `["roc_auc", "log_loss"]`).	required
`y_true`	`NDArray`	Ground truth binary target labels of shape (n_samples,).	required
`y_pred`	`NDArray or PredictionBundle`	Predicted probabilities or decision values of shape (n_samples,) or (n_samples, 2), or a `PredictionBundle` carrying both flat point predictions and a full predictive distribution (required for `predict_dist`-based metrics such as crps).	required
`n_bootstraps`	`int`	Number of bootstrap resampling iterations.	`1000`
`ci_level`	`float`	Confidence level for the calculated interval bounds (e.g., 0.95 for 95% CI).	`0.95`
`random_state`	`int, np.random.Generator, or None`	Seed or random generator instance to ensure reproducible resampling.	`None`
`n_jobs`	`int or None`	Number of parallel jobs used to compute the per-resample metrics (e.g. `ici`'s per-resample `SplineCalib` fits, or the pair-resampling fallback for `predict_dist`-based metrics without a `per_sample_func`). Not used for metrics with a `per_sample_func`.	`1`

Returns:

Name	Type	Description
`ci_results`	`dict[str, dict[str, float]]`	Dictionary mapping each metric key to a dictionary containing: `"point_estimate"`: Metric score computed on the original dataset. `"ci_lower"`: Lower confidence boundary. `"ci_upper"`: Upper confidence boundary.

Raises:

Type	Description
`ValueError`	If `ci_level` is not strictly between 0.0 and 1.0, if all bootstrap iterations fail due to severe class imbalance, if a `predict_dist`-based metric is requested without a distribution, or if the distribution and `y_true` have different numbers of samples.

Source code in medpipe/metrics/core.py

def bootstrap_confidence_intervals(
    metrics: list[str],
    y_true: npt.NDArray,
    y_pred: npt.NDArray | PredictionBundle,
    n_bootstraps: int = 1000,
    ci_level: float = 0.95,
    random_state: int | np.random.Generator | None = None,
    n_jobs: int | None = 1,
) -> dict[str, dict[str, float]]:
    """Compute non-parametric bootstrap confidence intervals for metrics.

    Resamples paired target labels and predictions with replacement, evaluating
    all specified metrics across bootstrap iterations. Edge cases where resamples
    lack sufficient class diversity (e.g., single-class bootstrap draws) are safely
    caught and skipped.

    Parameters
    ----------
    metrics : list[str]
        Metric identifier keys registered in `MetricRegistry`
        (e.g., `["roc_auc", "log_loss"]`).
    y_true : npt.NDArray
        Ground truth binary target labels of shape (n_samples,).
    y_pred : npt.NDArray or PredictionBundle
        Predicted probabilities or decision values of shape (n_samples,) or
        (n_samples, 2), or a `PredictionBundle` carrying both flat point
        predictions and a full predictive distribution (required for
        `predict_dist`-based metrics such as crps).
    n_bootstraps : int, default=1000
        Number of bootstrap resampling iterations.
    ci_level : float, default=0.95
        Confidence level for the calculated interval bounds (e.g., 0.95 for 95% CI).
    random_state : int, np.random.Generator, or None, default=None
        Seed or random generator instance to ensure reproducible resampling.
    n_jobs : int or None, default=1
        Number of parallel jobs used to compute the per-resample metrics
        (e.g. `ici`'s per-resample `SplineCalib` fits, or the pair-resampling
        fallback for `predict_dist`-based metrics without a
        `per_sample_func`). Not used for metrics with a `per_sample_func`.

    Returns
    -------
    ci_results : dict[str, dict[str, float]]
        Dictionary mapping each metric key to a dictionary containing:

        - ``"point_estimate"``: Metric score computed on the original dataset.
        - ``"ci_lower"``: Lower confidence boundary.
        - ``"ci_upper"``: Upper confidence boundary.

    Raises
    ------
    ValueError
        If ``ci_level`` is not strictly between 0.0 and 1.0, if all bootstrap
        iterations fail due to severe class imbalance, if a
        `predict_dist`-based metric is requested without a distribution, or
        if the distribution and ``y_true`` have different numbers of samples.

    """
    if not (0.0 < ci_level < 1.0):
        raise ValueError(f"ci_level must be between 0.0 and 1.0, got {ci_level}")

    y_true_arr = np.asarray(y_true).ravel()

    if isinstance(y_pred, PredictionBundle):
        bundle = y_pred
    else:
        y_pred_arr = np.asarray(y_pred)
        point = y_pred_arr[:, 1] if y_pred_arr.ndim == 2 else y_pred_arr.ravel()
        bundle = PredictionBundle(point=point, dist=None)

    _check_dist_length(y_true_arr, bundle.dist)

    rng = (
        random_state
        if isinstance(random_state, np.random.Generator)
        else np.random.default_rng(random_state)
    )

    alpha = (1.0 - ci_level) / 2.0
    lower_percentile = alpha * 100.0
    upper_percentile = (1.0 - alpha) * 100.0

    flat_metrics = [
        m for m in metrics if MetricRegistry.get(m).response_method != "predict_dist"
    ]
    dist_metrics = [
        m for m in metrics if MetricRegistry.get(m).response_method == "predict_dist"
    ]

    ci_results: dict[str, dict[str, float]] = {}

    if flat_metrics:
        if bundle.point is None:
            raise ValueError(
                "Point predictions are required for flat metrics, but the "
                "PredictionBundle has `point=None`."
            )
        ci_results.update(
            _bootstrap_flat_metrics(
                flat_metrics,
                y_true_arr,
                bundle.point,
                n_bootstraps,
                lower_percentile,
                upper_percentile,
                rng,
                n_jobs,
            )
        )

    if dist_metrics:
        if bundle.dist is None:
            raise ValueError(
                f"Distributional metrics {dist_metrics} require a "
                "PredictionBundle with `dist` set."
            )
        ci_results.update(
            _bootstrap_distributional_metrics(
                dist_metrics,
                y_true_arr,
                bundle.dist,
                n_bootstraps,
                lower_percentile,
                upper_percentile,
                rng,
                n_jobs,
            )
        )

    return ci_results

registry

MetricSpec `dataclass`

Encapsulates execution details and metadata for an evaluation metric.

Parameters:

Name	Type	Description	Default
`name`	`str`	Unique lookup identifier for the metric.	required
`func`	`Callable`	Underlying metric evaluation function.	required
`response_method`	`str or tuple of str`	Scikit-learn model prediction method required by the metric (e.g., 'predict', 'predict_proba').	required
`display_name`	`str`	Human-readable name used for visual displays and reports.	required
`sklearn_scorer_name`	`str or None`	Optional pre-registered scikit-learn scorer string key.	`None`
`needs_threshold`	`bool`	Whether raw predictions must be rounded to discrete class labels before being passed to `func` (e.g., accuracy, precision). Metrics that consume continuous values directly (e.g., rmse, mae, or any `predict_proba`-based metric) must leave this as False.	`False`
`per_sample_func`	`Callable or None`	For `predict_dist`-based metrics only: a function `(y, dist) -> array of shape (n_samples,)` returning the metric's per-sample values. Used by `bootstrap_confidence_intervals` to compute CIs by resampling precomputed per-sample scores rather than resampling (and re-slicing) the distribution object itself. Metrics without it fall back to resampling the (y, dist) pairs, which is exact for any metric but slower.	`None`

Source code in medpipe/metrics/registry.py

@dataclass(frozen=True)
class MetricSpec:
    """Encapsulates execution details and metadata for an evaluation metric.

    Parameters
    ----------
    name : str
        Unique lookup identifier for the metric.
    func : Callable
        Underlying metric evaluation function.
    response_method : str or tuple of str
        Scikit-learn model prediction method required by the metric
        (e.g., 'predict', 'predict_proba').
    display_name : str
        Human-readable name used for visual displays and reports.
    sklearn_scorer_name : str or None, default=None
        Optional pre-registered scikit-learn scorer string key.
    needs_threshold : bool, default=False
        Whether raw predictions must be rounded to discrete class labels
        before being passed to `func` (e.g., accuracy, precision). Metrics
        that consume continuous values directly (e.g., rmse, mae, or any
        `predict_proba`-based metric) must leave this as False.
    per_sample_func : Callable or None, default=None
        For `predict_dist`-based metrics only: a function `(y, dist) ->
        array of shape (n_samples,)` returning the metric's per-sample
        values. Used by `bootstrap_confidence_intervals` to compute CIs by
        resampling precomputed per-sample scores rather than resampling
        (and re-slicing) the distribution object itself. Metrics without it
        fall back to resampling the (y, dist) pairs, which is exact for any
        metric but slower.

    """

    name: str
    func: Callable
    response_method: str | tuple[str, ...]
    display_name: str
    sklearn_scorer_name: str | None = None
    needs_threshold: bool = False
    per_sample_func: Callable | None = None

    def get_scorer(self) -> Callable:
        """Construct a scikit-learn compatible scorer function for cross-validation.

        Returns
        -------
        scorer : Callable
            Scikit-learn scorer object suitable for model evaluation or tuning.
        """
        if self.response_method == "predict_dist":
            # sklearn's make_scorer only knows about predict/predict_proba/
            # decision_function, and none of those pass a full distribution
            # object to the metric function. Bypass it with a raw callable
            # matching sklearn's own scorer(estimator, X, y) -> float
            # convention instead.
            func = self.func

            def _distributional_scorer(estimator: Any, X: Any, y: Any) -> float:
                dist = estimator.predict_dist(X)
                # CRPS (and similar distributional losses) are lower-is-better,
                # so negate to match sklearn's "greater is better" scorer
                # convention (as sklearn's own neg_* scorers do).
                return -float(func(y, dist))

            return _distributional_scorer

        if self.sklearn_scorer_name:
            return get_scorer(self.sklearn_scorer_name)

        need_proba = "predict_proba" in (
            self.response_method
            if isinstance(self.response_method, tuple)
            else (self.response_method,)
        )
        return make_scorer(
            self.func,
            response_method="predict_proba" if need_proba else "predict",
        )

get_scorer

get_scorer() -> Callable

Construct a scikit-learn compatible scorer function for cross-validation.

Returns:

Name	Type	Description
`scorer`	`Callable`	Scikit-learn scorer object suitable for model evaluation or tuning.

Source code in medpipe/metrics/registry.py

def get_scorer(self) -> Callable:
    """Construct a scikit-learn compatible scorer function for cross-validation.

    Returns
    -------
    scorer : Callable
        Scikit-learn scorer object suitable for model evaluation or tuning.
    """
    if self.response_method == "predict_dist":
        # sklearn's make_scorer only knows about predict/predict_proba/
        # decision_function, and none of those pass a full distribution
        # object to the metric function. Bypass it with a raw callable
        # matching sklearn's own scorer(estimator, X, y) -> float
        # convention instead.
        func = self.func

        def _distributional_scorer(estimator: Any, X: Any, y: Any) -> float:
            dist = estimator.predict_dist(X)
            # CRPS (and similar distributional losses) are lower-is-better,
            # so negate to match sklearn's "greater is better" scorer
            # convention (as sklearn's own neg_* scorers do).
            return -float(func(y, dist))

        return _distributional_scorer

    if self.sklearn_scorer_name:
        return get_scorer(self.sklearn_scorer_name)

    need_proba = "predict_proba" in (
        self.response_method
        if isinstance(self.response_method, tuple)
        else (self.response_method,)
    )
    return make_scorer(
        self.func,
        response_method="predict_proba" if need_proba else "predict",
    )

MetricRegistry

Bases: BaseRegistry[MetricSpec]

Registry managing available and custom MetricSpec definitions.

Source code in medpipe/metrics/registry.py

class MetricRegistry(BaseRegistry[MetricSpec]):
    """Registry managing available and custom MetricSpec definitions."""

    @classmethod
    def register_spec(cls, spec: MetricSpec) -> MetricSpec:
        """Register a MetricSpec instance directly into the registry.

        Parameters
        ----------
        spec : MetricSpec
            The metric specification instance to register.

        Returns
        -------
        MetricSpec
            The registered metric specification instance.
        """
        cls._registry[spec.name] = spec
        return spec

    @classmethod
    def get(cls, name: str) -> MetricSpec:
        """Retrieve a registered MetricSpec instance by name.

        Parameters
        ----------
        name : str
            The name identifier of the metric to retrieve.

        Returns
        -------
        MetricSpec
            The matching metric specification instance.

        Raises
        ------
        ValueError
            If the requested metric name is not found in the registry.
        """
        if name in cls._registry:
            return cls._registry[name]

        raise ValueError(
            f"'{name}' was not found in available metrics. "
            f"Available metrics are {cls.list_registered()}"
        )

register_spec `classmethod`

register_spec(spec: MetricSpec) -> MetricSpec

Register a MetricSpec instance directly into the registry.

Parameters:

Name	Type	Description	Default
`spec`	`MetricSpec`	The metric specification instance to register.	required

Returns:

Type	Description
`MetricSpec`	The registered metric specification instance.

Source code in medpipe/metrics/registry.py

@classmethod
def register_spec(cls, spec: MetricSpec) -> MetricSpec:
    """Register a MetricSpec instance directly into the registry.

    Parameters
    ----------
    spec : MetricSpec
        The metric specification instance to register.

    Returns
    -------
    MetricSpec
        The registered metric specification instance.
    """
    cls._registry[spec.name] = spec
    return spec

get `classmethod`

get(name: str) -> MetricSpec

Retrieve a registered MetricSpec instance by name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name identifier of the metric to retrieve.	required

Returns:

Type	Description
`MetricSpec`	The matching metric specification instance.

Raises:

Type	Description
`ValueError`	If the requested metric name is not found in the registry.

Source code in medpipe/metrics/registry.py

@classmethod
def get(cls, name: str) -> MetricSpec:
    """Retrieve a registered MetricSpec instance by name.

    Parameters
    ----------
    name : str
        The name identifier of the metric to retrieve.

    Returns
    -------
    MetricSpec
        The matching metric specification instance.

    Raises
    ------
    ValueError
        If the requested metric name is not found in the registry.
    """
    if name in cls._registry:
        return cls._registry[name]

    raise ValueError(
        f"'{name}' was not found in available metrics. "
        f"Available metrics are {cls.list_registered()}"
    )

Metrics

Available metrics

Classification

Regression

Adding a metric

core

PredictionBundle dataclass

ici_score

crps_score

build_scorers

compute_metrics

bootstrap_confidence_intervals

registry

MetricSpec dataclass

get_scorer

MetricRegistry

register_spec classmethod

get classmethod

PredictionBundle `dataclass`

MetricSpec `dataclass`

register_spec `classmethod`

get `classmethod`