Reference

Evaluation

Task-grouped cross-validation, the paper's metrics and paired bootstrap intervals.

cross_validate is the only code in the library that decides what is training data. See Choose and evaluate a head and Leakage safety.

from tabagent import TabAgent, TabHead, cross_validate, datasets

name = "skillsbench/skill-select"
result = cross_validate(
    TabAgent(datasets.card(name), head=TabHead("ranking")),
    datasets.load(name, download=True),
    folds=datasets.folds(name),
)
print(result.summary(["p@r", "recall@9"]))

cross_validate

cross_validate(
    model: Model,
    table: DecisionTable,
    *,
    folds: Mapping | None = None,
    n_splits: int = 5,
    fold_seed: int = DEFAULT_SEED,
    seeds: Iterable[int] = PAPER_SEEDS,
    stratify: bool | str = False,
) -> CVResult

Grouped K-fold out-of-fold scoring of model on table (paper Sec. 4.3; App. M).

For every seed and fold the model is cloned with that seed, fitted on the training groups and scored on a label-free view of the held-out groups.

Parameters:

Name	Type	Description	Default
model	Model	anything with clone(seed), fit(table) and score(table), such as a TabAgent.	required
table	DecisionTable	the labeled decision table.	required
folds	Mapping | None	a pinned {group: fold} map (use the bundled paper folds to reproduce published numbers); otherwise folds are assigned with fold_seed.	None
n_splits	int	number of folds when folds is not given.	5
fold_seed	int	seed of the fold assignment.	DEFAULT_SEED
seeds	Iterable[int]	model seeds; the default is the paper's three seeds. Seeds never change the folds.	PAPER_SEEDS
stratify	bool | str	True stratifies by spec.strata and |G(t)| deciles; a column name overrides the stratum column.	False

Returns:

Type	Description
CVResult	out-of-fold scores (one column per seed) with metric and interval helpers.

CVResult dataclass

CVResult(table: DecisionTable, oof: DataFrame, folds: dict)

Out-of-fold scores for every seed, plus the metric and interval helpers of the paper protocol.

seeds property

seeds: list

The model seeds, one oof column each.

per_decision

per_decision(metric: str, **kw) -> Series

Metric per decision, averaged over seeds (paper: task-level scores averaged over seeds).

per_group

per_group(metric: str, **kw) -> Series

Metric per task: the mean over the task's decisions (one value when a task is one decision).

ci

ci(metric: str, level: str = 'group', **kw) -> Interval

Bootstrap interval of the metric's mean over tasks (level="group") or decisions; kw go to tabagent.evaluation.stats.bootstrap_ci.

summary

summary(metrics: Iterable[str] = ('p@r',), level: str = 'group', **kw) -> DataFrame

One row per metric: estimate, 95% interval bounds and the number of resampled units.

compare

compare(other: CVResult, metric: str, level: str = 'group', **kw) -> Interval

Paired bootstrap CI of (self - other) on the units both results cover.

cross_validate_discovery

cross_validate_discovery(
    trajectories,
    target: str,
    *,
    adapter: Any = "auto",
    llm: Any = None,
    config: Any = None,
    kind: str = "ranking",
    folds: Mapping | None = None,
    n_splits: int = 5,
    fold_seed: int = DEFAULT_SEED,
    seeds: Iterable[int] = (0,),
    search: str | None = "fast",
    iterations: int | None = None,
    run_dir: str | Path | None = None,
    trust_extractor: bool = False,
    sandbox: Any = None,
    redact: Callable[[dict], dict] | None = None,
    fold_workers: int = 3,
) -> DiscoveryCVResult

Grouped cross-validation of the whole onboarding pipeline, with TabSchema discovery run inside every training fold (see the module docstring).

Parameters:

Name	Type	Description	Default
trajectories		a file or directory of agent logs, or a list of Trajectory.	required
target	str	the decision component to replace (as onboard's decision).	required
adapter	Any	the log adapter, or "auto" to detect it from the files.	'auto'
llm	Any	the TabSchema LLM (default auto_llm(config.model), from environment keys) and	None
config	Any	the TabSchema LLM (default auto_llm(config.model), from environment keys) and TabSchemaConfig. config.n_analyze applies to each fold's discovery, so the LLM cost is about n_splits times one onboarding; config.max_cost_usd (with config.prices) caps all folds together.	None
kind	str	ranking, multilabel or single.	'ranking'
folds	Mapping | None	as for cross_validate; the defaults give the folds of onboard's	None
n_splits	int	as for cross_validate; the defaults give the folds of onboard's	5
fold_seed	int	as for cross_validate; the defaults give the folds of onboard's report, so the two estimates are comparable.	DEFAULT_SEED
seeds	Iterable[int]	head seeds (onboard's report uses one); discovery and recipe selection run once per fold.	(0,)
search	str | None	the recipe search of tabagent.auto.select (fast or full) run on each training fold, as onboard does on all data; None fits the paper's CatBoost preset without TabSynth.	'fast'
iterations	int | None	override the CatBoost iteration count (for quick runs).	None
run_dir	str | Path | None	the directory of the per-fold run directories (default a new one under $TABAGENT_CACHE/runs); a rerun with the same directory and folds makes no new LLM calls.	None
trust_extractor	bool	run each fold's extractor in-process instead of in the sandbox.	False
sandbox	Any	the sandbox for discovery and extraction (default SubprocessSandbox; DockerSandbox() when the logs or the code are untrusted).	None
redact	Callable[[dict], dict] | None	applied to every raw trajectory before anything can reach the LLM provider.	None
fold_workers	int	folds whose discovery runs at the same time (each makes up to config.workers LLM calls at once). With config.max_cost_usd, each concurrent fold gets an equal share of the budget.	3

Returns:

Type	Description
DiscoveryCVResult	out-of-fold scores with the usual metric helpers (summary, ci, compare), each fold's discovery, the recipe chosen per fold and the total LLM cost.

assign_folds

assign_folds(
    groups: Iterable[Hashable],
    n_splits: int = 5,
    seed: int = DEFAULT_SEED,
    strata: Mapping[Hashable, Hashable] | None = None,
) -> dict

Map each unique group to a fold in range(n_splits).

The unique groups are sorted (so folds do not depend on row order), shuffled with default_rng(seed) and dealt round-robin. With strata the groups of each stratum are shuffled and dealt round-robin, continuing the deal across strata, so each fold gets a near-equal share of every stratum (the paper's stratification by application and |G(t)| decile).

per_decision

per_decision(table: DecisionTable, scores: ndarray, metric: str, **kw) -> Series

Dispatch a per-decision metric by name: p@r, recall@K, hit@K, route@1, accuracy, ml_precision, ml_recall, ml_f1. Keyword arguments (tie_break, threshold) go to the metric; accuracy takes none, since a single-class prediction has no ties.

bootstrap_ci

bootstrap_ci(
    values,
    *,
    n_resamples: int = DEFAULT_RESAMPLES,
    confidence: float = 0.95,
    seed: int = DEFAULT_SEED,
    method: str = "percentile",
) -> Interval

Bootstrap confidence interval of the mean of per-unit values (one value per task or decision).

NaN values are dropped. method is "percentile" (default) or "BCa"; see the module docstring.

paired_bootstrap

paired_bootstrap(a, b, **kw) -> Interval

CI of mean(a - b) over paired units (the paper's paired task-level bootstrap, 2,000 resamples).

Units where either value is NaN are dropped. An improvement is claimed only when the interval excludes 0 (Interval.excludes_zero).

Interval dataclass

Interval(estimate: float, low: float, high: float, method: str, n: int, confidence: float = 0.95)

A point estimate with its confidence interval, the method that produced it and the number of units.

excludes_zero property

excludes_zero: bool

True when the interval lies strictly above or below zero (the paper's significance rule).

holm

holm(pvalues) -> ndarray

Holm-Bonferroni adjusted p-values (monotone step-down), in the input order. The paper controls the family-wise error of a metric across applications this way (App. M).