Skip to content

Examples

The scripts below are in the examples/ folder of the repository, next to the configuration files they use. Run them from that folder, since the paths in the configurations are relative to where a script is run:

cd examples
python manual_workflow.py

Every run saves its models, results and figures in a new artifacts/vN folder. The examples use the fast run mode so that they finish quickly. See the run modes for the other modes.

Run an experiment

The minimal examples create the pipeline from a TOML file and run it. They are described in the quickstart.

Go through the steps yourself

run() prepares the data, fits the models, evaluates them and draws the figures. Each step can also be called on its own, for example to look at the predictions or to evaluate the models on other data. The configuration of the example is changed in Python before it is validated, to use the fast run mode.

manual_workflow.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import tomllib

from medpipe import MedpipeClassifier, MedpipeClassifierConfig

# Load the configuration of the example and only fit and evaluate the models
with open("default_classifier_config.toml", "rb") as file:
    raw_config = tomllib.load(file)
raw_config["meta"]["run_mode"] = "fast"
config = MedpipeClassifierConfig.model_validate(raw_config)

pipe = MedpipeClassifier(config)

# Load the data and split it as described in the configuration
X_train, y_train, X_recal, y_recal, X_test, y_test, groups_train = pipe.prepare_data()

# Fit one model per outcome
pipe.fit(X_train, y_train, X_recal, y_recal, groups_train)

for outcome in pipe.models:
    # Predict the probability of each class for the test set
    probas = pipe.predict_proba(X_test, outcome=outcome)

    # Evaluate the model, overall and for the subgroups of SEX
    results = pipe.evaluate(
        X_test, y_test[outcome], outcome=outcome, subgroup_specs={"SEX": "SEX"}
    )
    auroc = results["overall"]["roc_auc"]
    print(f"{outcome}: AUROC {auroc['point_estimate']:.2f}")

    # Draw the figures for the outcome
    pipe.plot_all(y_test[outcome].to_numpy(), probas, outcome=outcome, n_bootstraps=50)
  • prepare_data() loads the dataset and splits it as described in [workflow.validation]. The recalibration set is None when no recalibration split is defined.
  • fit() fits one model per outcome, and models holds them by outcome name.
  • evaluate() returns the metrics with their confidence intervals, overall and for the subgroups it is given. The results are also saved in the results/ folder of the run.
  • plot_all() draws the figures of an outcome. The settings of [display] are used, and the arguments given here take precedence.

Reuse a fitted pipeline

The folder of a run holds everything needed to restore the pipeline: the resolved configuration and the fitted models. load() returns a pipeline that is ready to predict and evaluate. What it saves from then on goes in an eval folder inside the folder of the run, so the original results are not overwritten.

load_saved_run.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import tomllib

from medpipe import MedpipeClassifier, MedpipeClassifierConfig

with open("default_classifier_config.toml", "rb") as file:
    raw_config = tomllib.load(file)
raw_config["meta"]["run_mode"] = "fast"

# Run the pipeline once. The models are saved in the folder of the run.
pipe = MedpipeClassifier(MedpipeClassifierConfig.model_validate(raw_config))
pipe.run()
run_dir = pipe.run_dir

# Later, restore the pipeline from the folder of the run
restored = MedpipeClassifier.load(run_dir)

# Use the fitted models on the test set
_, _, _, _, X_test, y_test, _ = restored.prepare_data()
for outcome in restored.models:
    predictions = restored.predict(X_test, outcome=outcome)
    print(f"{outcome}: {int(predictions.sum())} positive predictions")

Write the configuration in Python

A configuration does not need a TOML file. A dictionary with the same structure is validated in the same way, and the same errors are raised. This is convenient to generate configurations, for example to compare several algorithms. base_artifact_dir chooses where the runs are saved, and verbose_override changes the console output without editing the configuration.

config_in_python.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
from medpipe import MedpipeClassifier, MedpipeClassifierConfig

# A configuration can be written in Python instead of a TOML file
config = MedpipeClassifierConfig.model_validate(
    {
        "meta": {"project_name": "python_config_demo", "run_mode": "fast"},
        "data": {
            "path": "../tests/test_data.csv",
            "predictors": ["AGE", "SEX", "OP_SEVERITY"],
            "outcomes": ["MORTALITY_30D"],
        },
        "default_model": {
            "algorithm": "LogisticRegression",
            "hyperparameters": {"max_iter": 500},
        },
        "workflow": {
            "preprocessing": {
                "preprocess": True,
                "operations": [
                    {"name": "OrdinalEncoder", "columns": ["SEX"]},
                    {"name": "StandardScaler", "columns": ["AGE", "OP_SEVERITY"]},
                ],
            },
            "validation": {
                "test_split": {"strategy": "random", "test_size": 0.3},
            },
            "evaluation": {
                "metrics": {"metrics": ["roc_auc", "brier_score"], "n_bootstraps": 100},
            },
        },
    }
)

# The artifacts are saved in a custom folder and the console shows all messages
pipe = MedpipeClassifier(
    config, base_artifact_dir="python_config_runs", verbose_override="info"
)
results = pipe.run()
print(results["evaluations"]["MORTALITY_30D"]["overall"]["brier_score"])

Add your own model, preprocessing operation or metric

The registries make a component available by name in the configuration. They are described with the models, the preprocessing operations and the metrics of the API reference. This example registers one of each and uses them in a configuration.

custom_components.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
from functools import partial

from sklearn.base import BaseEstimator, OneToOneFeatureMixin, TransformerMixin
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import fbeta_score
from sklearn.utils.validation import validate_data

from medpipe import (
    MedpipeClassifier,
    MedpipeClassifierConfig,
    MetricRegistry,
    ModelRegistry,
)
from medpipe.data import PreprocessorRegistry
from medpipe.metrics import MetricSpec


# A model, registered under the name used in the configuration
@ModelRegistry.register("ShallowForest")
class ShallowForest(RandomForestClassifier):
    def __init__(self, n_estimators=100, max_depth=3, random_state=None):
        super().__init__(
            n_estimators=n_estimators, max_depth=max_depth, random_state=random_state
        )


# A preprocessing operation. The name defaults to the name of the class.
# OneToOneFeatureMixin gives it the feature names that the pipeline needs to
# pass the data from one operation to the next.
@PreprocessorRegistry.register()
class ClipOutliers(OneToOneFeatureMixin, TransformerMixin, BaseEstimator):
    def __init__(self, lower=0.0, upper=1.0):
        self.lower = lower
        self.upper = upper

    def fit(self, X, y=None):
        validate_data(self, X)  # Records the names of the columns
        return self

    def transform(self, X):
        return X.clip(self.lower, self.upper)


# A metric that weights recall twice as much as precision
MetricRegistry.register_spec(
    MetricSpec(
        name="f2",
        func=partial(fbeta_score, beta=2),
        response_method="predict",
        display_name="F2",
        needs_threshold=True,
    )
)

# The components are registered before the configuration is validated
config = MedpipeClassifierConfig.model_validate(
    {
        "meta": {"project_name": "custom_components_demo", "run_mode": "fast"},
        "data": {
            "path": "../tests/test_data.csv",
            "predictors": ["AGE", "OP_SEVERITY"],
            "outcomes": ["ANY_COMP"],
        },
        "default_model": {
            "algorithm": "ShallowForest",
            "hyperparameters": {"max_depth": 4, "random_state": 42},
        },
        "workflow": {
            "preprocessing": {
                "preprocess": True,
                "operations": [
                    {
                        "name": "ClipOutliers",
                        "columns": ["AGE"],
                        "lower": 18,
                        "upper": 90,
                    },
                ],
            },
            "validation": {"test_split": {"strategy": "random", "test_size": 0.3}},
            "evaluation": {
                "metrics": {"metrics": ["roc_auc", "f2"], "n_bootstraps": 100}
            },
        },
    }
)

pipe = MedpipeClassifier(config)
results = pipe.run()
overall = results["evaluations"]["ANY_COMP"]["overall"]
auroc = overall["roc_auc"]["point_estimate"]
f2 = overall["f2"]["point_estimate"]
print(f"AUROC {auroc:.2f}, F2 {f2:.2f}")
  • The components have to be registered before the pipeline is created. The model and the metric names are checked when the configuration is validated, and the preprocessing operation is looked up when the data is prepared.
  • A preprocessing operation has to keep the names of its columns, which OneToOneFeatureMixin provides, as explained with the preprocessing operations.

Predict a distribution

For a regression, OrdBoostRegressor predicts a distribution of the outcome for each patient instead of a single value. The distribution is used by the crps metric and by the figures that check whether the prediction intervals are reliable. The prediction methods are described on the regression page.

regression_workflow.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import tomllib

from medpipe import MedpipeRegressor, MedpipeRegressorConfig

with open("default_regressor_config.toml", "rb") as file:
    raw_config = tomllib.load(file)
raw_config["meta"]["run_mode"] = "fast"
pipe = MedpipeRegressor(MedpipeRegressorConfig.model_validate(raw_config))

X_train, y_train, X_recal, y_recal, X_test, y_test, groups_train = pipe.prepare_data()
pipe.fit(X_train, y_train, X_recal, y_recal, groups_train)

for outcome in pipe.models:
    y_true = y_test[outcome].to_numpy()

    # Point predictions, and the distribution of the outcome for each patient
    predictions = pipe.predict(X_test, outcome=outcome)
    distribution = pipe.predict_dist(X_test, outcome=outcome)

    # Evaluate the model with the metrics of the configuration
    results = pipe.evaluate(X_test, y_true, outcome=outcome)
    rmse = results["overall"]["rmse"]["point_estimate"]
    print(f"{outcome}: RMSE {rmse:.2f}")

    # Draw the coverage, sharpness, Winkler score, calibration and PIT figures
    pipe.plot_all(y_true, distribution, outcome=outcome, n_bootstraps=50)