nonconformist.evaluation.cross_val_score

nonconformist.evaluation.cross_val_score(model, x, y, iterations=10, folds=10, fit_params=None, scoring_funcs=None, significance_levels=None, verbose=False)

Evaluates a conformal predictor using cross-validation.

Parameters:

model : object

Conformal predictor to evaluate.

x : numpy array of shape [n_samples, n_features]

Inputs of data to use for evaluation.

y : numpy array of shape [n_samples]

Outputs of data to use for evaluation.

iterations : int

Number of iterations to use for evaluation. The data set is randomly shuffled before each iteration.

folds : int

Number of folds to use for evaluation.

fit_params : dictionary

Parameters to supply to the conformal prediction object on training.

scoring_funcs : iterable

List of evaluation functions to apply to the conformal predictor in each fold. Each evaluation function should have a signature scorer(prediction, y, significance).

significance_levels : iterable

List of significance levels at which to evaluate the conformal predictor.

verbose : boolean

Indicates whether to output progress information during evaluation.

Returns:

scores : pandas DataFrame

Tabulated results for each iteration, fold and evaluation function.