Reference

Onboarding and heads¶

Turn agent logs into a head, and load a saved one.

onboard is the one-call path from agent logs to a deployable head. load reads a saved head back, from a local directory or from the Hugging Face Hub. A TabAgentHead is the fitted model plus the TabSchema extractor that feeds it: the drop-in replacement for one agent decision.

import tabagent

head = tabagent.onboard("./agent_logs", decision="ToolShortlist")
print(head.report)  # grouped cross-validation vs BM25 and chance, latency, cost per decision
head.save("my_head")
head.decide(prefix, candidates, k=5)  # {"candidates", "scores", "confidence", "deferred", "error"}

head = tabagent.load("my_head", trust_extractor=True)  # trust only an extractor you have reviewed

onboard ¶

onboard(
    logs: str | Path | list[Trajectory],
    decision: str | None = None,
    *,
    adapter: Any = "auto",
    llm: LLM | None = None,
    model: str | None = None,
    kind: str | None = None,
    out: str | Path | None = None,
    n_analyze: int | None = None,
    max_cost_usd: float | None = None,
    prices: dict[str, tuple[float, float]] | None = None,
    dry_run: bool = False,
    redact: Callable[[dict], dict] | None = None,
    search: str = "fast",
    n_splits: int = 5,
    llm_cost_per_call: float | None = None,
    trust_extractor: bool = False,
    run_dir: str | Path | None = None,
    config: TabSchemaConfig | None = None,
    gate_coverage: float | None = None,
    iterations: int | None = None,
    sandbox: Any = None,
    stage_models: dict[str, str] | None = None,
    fallback_model: str | None = None,
    lazy: bool | None = None,
    on_other_formats: str = "skip",
) -> TabAgentHead | DryRun

Onboard an agent decision component from its logs (see the module docstring).

Parameters:

Name	Type	Description	Default
`logs`	`str \| Path \| list[Trajectory]`	a file or directory of agent logs, or a list of `Trajectory`.	required
`decision`	`str \| None`	the component to replace (see `tabagent components`); optional when the logs have one.	`None`
`adapter`	`Any`	the log adapter, or `"auto"` to detect it from the files.	`'auto'`
`llm`	`LLM \| None`	the TabSchema LLM, or the model name to build one from environment keys (default GPT-5.4).	`None`
`model`	`str \| None`	the TabSchema LLM, or the model name to build one from environment keys (default GPT-5.4).	`None`
`kind`	`str \| None`	the decision pattern: `ranking`, `multilabel` or `single`. Default: the adapter's pattern for the decision (`adapter.decision_kinds`: `multilabel` for CUGA's app analyzer), else `ranking`.	`None`
`out`	`str \| Path \| None`	save the head to this directory.	`None`
`n_analyze`	`int \| None`	decision points analyzed by the LLM (default 20).	`None`
`max_cost_usd`	`float \| None`	stop TabSchema when its LLM cost exceeds this. Needs a known price for the model (`prices`); a budget without a price raises `ValueError` before any LLM call.	`None`
`prices`	`dict[str, tuple[float, float]] \| None`	`{model: (USD per 1M input tokens, USD per 1M output tokens)}`.	`None`
`dry_run`	`bool`	return a `DryRun` (work and cost estimate) without calling the LLM.	`False`
`redact`	`Callable[[dict], dict] \| None`	applied to every raw trajectory before anything can reach the LLM provider (it gets a copy; a run read lazily stays lazy and is redacted each time it is read, so `redact` must be deterministic).	`None`
`search`	`str`	`fast` or `full` recipe search (`tabagent.auto.select`).	`'fast'`
`n_splits`	`int`	folds of the grouped cross-validation in the report.	`5`
`llm_cost_per_call`	`float \| None`	the replaced LLM call's cost, to report the savings.	`None`
`trust_extractor`	`bool`	run the LLM-written extractor in-process. Off by default: the extractor then runs in the sandbox, since trajectories are untrusted text that reaches the LLM writing the code.	`False`
`sandbox`	`Any`	where the LLM-written code runs, in discovery, in the decision table and in the returned head: `"subprocess"` (the default, `SubprocessSandbox`), `"docker"` or `"docker:<image>"` (`DockerSandbox`: no network, read-only filesystem), or a sandbox instance. Not with `trust_extractor`.	`None`
`run_dir`	`str \| Path \| None`	the TabSchema run directory itself (default: a new one under `$TABAGENT_CACHE/runs`). A rerun with the same `run_dir` resumes from it: answers it already holds cost nothing.	`None`
`stage_models`	`dict[str, str] \| None`	a model per TabSchema stage, e.g. `{"extractor": "azure/gpt-6-sol"}` (see `TabSchemaConfig.stage_models`); each needs a price in `prices` with a budget.	`None`
`fallback_model`	`str \| None`	asked once, for any stage, when the provider's content filter stops an answer.	`None`
`lazy`	`bool \| None`	read the runs again from their files when they are needed instead of holding them all (default: from 256 MB of logs); `False` holds them in memory (CLI `--no-lazy`). A lazily read run whose file changes stops the onboarding with `LogChangedError`.	`None`
`on_other_formats`	`str`	with `adapter="auto"`, `"skip"` runs of another log format in the folder with a warning (the default), or `"error"` to stop with `MixedFormatsError` (CLI `--on-other-formats`).	`'skip'`
`config`	`TabSchemaConfig \| None`	a full `TabSchemaConfig`; it then also sets `model`, `n_analyze`, `max_cost_usd`, `prices`, `stage_models` and `fallback_model`, which must not be passed separately.	`None`
`gate_coverage`	`float \| None`	calibrate the confidence gate so this share of decisions stays with the head (rankings).	`None`
`iterations`	`int \| None`	override the CatBoost iteration count (for quick runs).	`None`

Returns:

Type	Description
`TabAgentHead \| DryRun`	The fitted `TabAgentHead`, or a `DryRun` with `dry_run=True`. Before any LLM call (and in a dry run) the data is checked: some trajectories must be successful, the decision must occur at least twice in at least 2 task groups, and the adapter must give its labels and candidates; otherwise `ValueError` (or the adapter's `NotImplementedError`) says why.

load ¶

load(
    path: str | Path, trust_extractor: bool = False, trust_code: bool = False, **head_options: Any
) -> TabAgentHead | TabAgent

Load a saved head: a TabAgentHead for a directory written by onboard (it has head.json), else the TabAgent artifact written by TabAgent.save.

path is a local directory or a Hugging Face Hub repository, hf://OWNER/REPO[@REVISION], which is downloaded first (pip install "tabagent[hub]"; see tabagent.hub) and then loaded and verified exactly as a local directory.

A TabAgentHead decides from a trajectory prefix (decide(prefix, candidates, k=...)); a bare TabAgent decides from candidate rows (decide(rows, k=...)). trust_extractor, trust_code and the head_options (sandbox, redact, adapter) are described in TabAgentHead.load; a TabAgent has no extractor and takes no head_options.

TabAgentHead `dataclass` ¶

TabAgentHead(
    agent: TabAgent,
    target: str,
    extractor_code: str | None = None,
    report: Any = None,
    discovery: DiscoveryReport | None = None,
    gate_threshold: float | None = None,
    trust_extractor: bool = False,
    adapter: Any = None,
    sandbox: Any = None,
    redact: Callable[[dict], dict] | str | None = None,
    extractor_tests: dict | None = None,
    allow_unchecked_extractor: bool = False,
    relative_features: bool | None = None,
)

A fitted TabAgent with the TabSchema extractor that feeds it: the drop-in replacement for one agent decision.

decide(prefix, candidates) runs the extractor on the trajectory prefix, builds one row per candidate and returns the head's decision; decide(run=..., candidates=...) first builds the prefix from the live run with the adapter the head was trained with (live_prefix). An extractor with a per-candidate function (extract_candidate_features) runs once per decision, on the prefix and on every candidate, in one sandbox call, exactly as for the training rows. Heads come from onboard and are persisted with save and load.

Attributes:

Name	Type	Description
`agent`	`TabAgent`	the fitted `TabAgent`.
`target`	`str`	the replaced decision component (for example `ShortlisterAgent`).
`extractor_code`	`str \| None`	the compiled TabSchema extractor (`extract.py`), or None.
`report`	`Any`	the onboarding evaluation report (`tabagent.report.Report`).
`discovery`	`DiscoveryReport \| None`	the TabSchema discovery report.
`gate_threshold`	`float \| None`	decisions with a lower confidence are marked `deferred` (App. N.3).
`trust_extractor`	`bool`	run the extractor in-process (fast; for reviewed code) instead of in the sandbox.
`adapter`	`Any`	the log adapter of the training trajectories (an adapter or its `describe_adapter` description); `live_prefix` builds live prefixes with it, and `save` records it in `head.json`.
`sandbox`	`Any`	where an untrusted extractor runs: `"warm"` (the default, also `None`, or `"warm:<N>"` for N workers: `WarmSandbox`, persistent isolated interpreters that start once and serve every decision), `"subprocess"` (`SubprocessSandbox`, a fresh interpreter per decision), `"docker"` or `"docker:<image>"` (`DockerSandbox`: no network, read-only filesystem), or a sandbox instance. Workers of a warm pool start at the first decision (`warm()` starts them all now) and stop with `close()`, when the head is garbage-collected or when the interpreter exits. A sandbox built from a name re-applies the security-relevant static checks of the compile step to the extractor and never runs one that fails them (its decisions are deferred), unless `allow_unchecked_extractor`; a sandbox instance runs what it is given.
`redact`	`Callable[[dict], dict] \| str \| None`	the function applied to each training trajectory before discovery, as a callable or a `"module:function"` import path; `live_prefix` applies it to a live run too.
`extractor_tests`	`dict \| None`	the extractor's golden tests (`extractor_tests.json`: snapshots of onboarding prefixes, after `redact`, with the features they give), which `tabagent.verify_head` re-runs.
`allow_unchecked_extractor`	`bool`	run, still in the sandbox, an extractor that fails those static checks (a head saved before they were re-applied at serve time, whose `extract.py` imports `time`, say), with a warning. Review the extractor first: the sandbox is not a security boundary.
`relative_features`	`bool \| None`	whether the recipe search kept the within-decision relative features (`card.with_relative`; then they are in the fitted card); None when it did not try them (a single-class decision, `TabSchemaConfig(relative_features=False)`, or a head saved before). `head.json` records it.

decide ¶

decide(
    prefix: dict[str, Any] | None = None,
    candidates=None,
    k: int | None = None,
    defer_below: float | None = None,
    *,
    run: dict[str, Any] | None = None,
    trace: dict[str, Any] | None = None,
) -> dict[str, Any]

Score one live decision.

Parameters:

Name	Type	Description	Default
`prefix`	`dict[str, Any] \| None`	the trajectory before the decision, normalized by the adapter the head was onboarded with (what the adapter's `decision_points` put in `DecisionPoint.prefix`; `live_prefix` builds it from a live run).	`None`
`candidates`		the legal candidates: ids, `{"id"/"name", "description", "args"}` dicts or `Candidate` objects. Not used by single-class heads, which choose among their training classes.	`None`
`k`	`int \| None`	return the top `k` of a ranking.	`None`
`defer_below`	`float \| None`	confidence threshold of the gate; defaults to the calibrated `gate_threshold`.	`None`
`run`	`dict[str, Any] \| None`	instead of `prefix`, the live run so far in the format of the training logs, with its run-level task fields; the prefix is built with `live_prefix`.	`None`
`trace`	`dict[str, Any] \| None`	a dict that receives the decision's timings in seconds (`prefix_s`, `extraction_s`, `scoring_s`), the kind of the extractor's failure (`error_kind`: `ExecResult.kind`, or `no_extractor`; None when it succeeded) and its feature frame (`frame`, the head's input), for monitoring.	`None`

Returns:

Type Description

dict[str, Any]

{"candidates": [ids], "scores": {id: p}, "confidence": c, "deferred": bool, "error": None}, or for a single-class head {"choice": label, "scores": {label: p}, "confidence": p(choice), "deferred": bool, "error": None}. Tied scores are ordered by candidate id. When the extractor fails on the prefix, its features are empty (None, as for a training decision point it failed on), the decision is deferred to the original LLM whatever its confidence, and error says why.

live_prefix ¶

live_prefix(run: dict[str, Any], adapter: Any = None) -> dict[str, Any]

The prefix of a live decision, built from run (the run so far, in the format of the training logs, with its run-level task fields) exactly as the training prefixes were: redact first, then the head's adapter (or adapter; see tabagent.adapters.live_prefix).

warm ¶

warm() -> int

Start the extractor's warm workers now rather than at the first decision (a serving process calls this at start-up). Returns the number of running workers (0 without a warm pool). Raises ExtractorBrokenError when the extractor cannot run at all (it does not compile, or breaks the extraction rules).

close ¶

close() -> None

Stop the extractor's warm workers, if any. A later decision starts a new pool.

calibrate_gate ¶

calibrate_gate(cv_result, target_coverage: float = 0.9) -> float

Set gate_threshold so that target_coverage of the cross-validated decisions stay with the head and the least confident 1 - target_coverage are deferred to the LLM (App. N.3). Returns the threshold. Rankings only: a cross-validated single-class head keeps no scores to calibrate on.

save ¶

save(path: str | Path) -> Path

Write the head to the directory path: the model artifact (model/), extract.py and its golden tests (extractor_tests.json, trajectory snippets), head.json (with the adapter and its prefix settings, the redaction function's import path, when known, and the paper's citation) and, when present, the evaluation and discovery reports. A card that pins no hash for the extractor gets the hash of extract.py before it is written, so load can check the code.

load `classmethod` ¶

load(
    path: str | Path,
    trust_extractor: bool = False,
    trust_code: bool = False,
    *,
    sandbox: Any = None,
    redact: Callable[[dict], dict] | str | None = None,
    adapter: Any = None,
    allow_unpinned: bool = False,
    allow_unchecked_extractor: bool = False,
) -> TabAgentHead

Load a saved head from a directory or hf://OWNER/REPO[@REVISION] (tabagent.hub). extract.py must match the hash recorded in the card. A card that records no hash (a head saved before the hash was always pinned) is refused, since its code could not be checked; after reviewing extract.py, allow_unpinned=True loads it with a warning, and saving the head again pins it.

trust_extractor=True runs the extractor in-process (fast); only do this for an extractor you have reviewed. Otherwise it runs in sandbox (see the class attributes). trust_code allows card features computed by modules outside tabagent. adapter overrides the adapter recorded in head.json. redact is never imported from head.json: when the head records one and none is passed, a warning says so, since live runs then reach the extractor unredacted. allow_unchecked_extractor is described in the class attributes.

features_for ¶

features_for(prefix: dict[str, Any]) -> dict[str, Any]

The extractor's features for one trajectory prefix: every extractor column of the card, None where the extractor gave no value or failed (as in training).

candidate_features_for ¶

candidate_features_for(prefix: dict[str, Any], candidates) -> list[dict[str, Any]]

The per-candidate features of a decision: for each candidate (as decide takes them), every per-candidate column of the card, None where the extractor gave no value or failed (as in training).

DryRun `dataclass` ¶

DryRun(
    n_trajectories: int,
    n_successful: int,
    n_decision_points: int,
    n_analyzed: int,
    est_input_tokens: int,
    est_output_tokens: int,
    est_cost_usd: float | None,
    components: dict[str, int] = dict(),
    model: str = DEFAULT_MODEL,
    problems: list[str] = list(),
    stages: dict[str, dict[str, Any]] = dict(),
    warnings: list[str] = list(),
)

What discovery would process, an estimate of its size and cost per stage (stages; tabagent.tabschema.estimate), and the problems that would stop a real run before any LLM call (problems).

RequestError ¶

Bases: ValueError

A decision request the head cannot answer as given (the HTTP service answers it with status 400): both a prefix and a run, a run the head cannot turn into a prefix, a ranking head without candidates, a malformed candidate. A failure of the head itself is not one.