Reference

Logs and adapters¶

Find what is worth replacing, diagnose the logs, and read them into trajectories.

Both scan and check_logs read the logs only and call no LLM. The guide is Check and debug logs.

import tabagent

print(tabagent.scan("./agent_logs").summary())
check = tabagent.check_logs("./agent_logs", decision="ToolShortlist")
print(check.summary(), check.ok)

scan ¶

scan(
    logs: str | Path | list[Trajectory],
    *,
    adapter: Any = "auto",
    prices: Prices | None = None,
    top: int = 3,
    lazy: bool | None = None,
    on_other_formats: str = "skip",
) -> ScanReport

Rank the components of an agent's logs that call an LLM by what replacing them could save.

Parameters:

Name	Type	Description	Default
`logs`	`str \| Path \| list[Trajectory]`	a file or directory of agent logs, or a list of `Trajectory` (then pass `adapter`).	required
`adapter`	`Any`	the log adapter, or `"auto"` to detect it from the files (as `onboard` does).	`'auto'`
`prices`	`Prices \| None`	`(USD per 1M input tokens, USD per 1M output tokens)` for every call, or `{model: (in, out)}` by the model a call records. Calls are then priced from their recorded tokens instead of a recorded cost; without prices, a cost the logs record is used.	`None`
`top`	`int`	how many closed-set candidates the summary details, with their next step.	`3`
`lazy`	`bool \| None`	read runs again from their files when needed instead of holding them (default: from 256 MB of logs).	`None`
`on_other_formats`	`str`	with `adapter="auto"`, `"skip"` runs of another log format with a warning, or `"error"` (`MixedFormatsError`).	`'skip'`

Returns:

Type	Description
`ScanReport`	A `ScanReport`: `summary()`, `to_markdown()`, `to_dict()`, `to_json()`.

ScanReport `dataclass` ¶

ScanReport(
    logs: str,
    adapter: str,
    n_runs: int,
    n_successful: int,
    n_calls: int,
    ranked_by: str,
    cost_source: str | None,
    components: list[ComponentScan],
    notes: list[str] = list(),
    prices: Any = None,
    top: int = 3,
)

The ranked components of a scan: summary(), to_markdown(), to_json().

Attributes:

Name	Type	Description
`components`	`list[ComponentScan]`	every component of the agent that calls an LLM, ranked by `ranked_by` (cost, else tokens, latency or calls, whichever the logs record first). Each is a `ComponentScan` with a verdict (`closed-set`, `unknown`, `open values` or `free text`), its evidence and the `onboard` command that replaces it.
`candidates`	`list[ComponentScan]`	the closed-set components, best first.
`n_runs`	`int`	the scanned runs, the successful ones and the LLM calls.
`n_successful`	`int`	the scanned runs, the successful ones and the LLM calls.
`n_calls`	`int`	the scanned runs, the successful ones and the LLM calls.
`notes`	`list[str]`	how the logs record usage, and what could not be counted.

Examples:

import tabagent

report = tabagent.scan("./agent_logs", prices=(PRICE_IN, PRICE_OUT))
print(report.summary())
report.candidates[0].next_steps  # the onboard command for the top closed-set component

candidates `property` ¶

candidates: list[ComponentScan]

The closed-set components, best first.

to_dict ¶

to_dict() -> dict[str, Any]

The report as a JSON-able dict.

to_json ¶

to_json(indent: int | None = 1) -> str

The report as JSON (see to_dict).

summary ¶

summary(width: int = 100) -> str

The report as plain text for a terminal, prose wrapped at width characters.

to_markdown ¶

to_markdown() -> str

The report as Markdown.

check_logs ¶

check_logs(
    logs: str | Path,
    *,
    adapter: Any = "auto",
    decision: str | None = None,
    sample: int = 200,
    lazy: bool | None = None,
    on_other_formats: str = "skip",
) -> LogCheck

Diagnose agent logs before an onboarding; reads the logs only, calls no LLM.

Reports which files were read, skipped or repaired (and why), the runs and their outcomes and, for the decision (default: every decision the adapter builds candidates and labels for), its decision points, candidates and labels (chosen candidates that are not among the offered ones, with near-miss names), the train/serve parity of its prefixes on sample decision points (spread evenly; all of them when there are fewer), and the adapter's check that no prefix holds post-decision data. LogCheck.ok is False when an onboarding would stop; every finding says what is wrong, where and how to fix it. lazy and on_other_formats are read as by onboard. CLI: tabagent check-logs LOGS [--decision NAME].

LogCheck `dataclass` ¶

LogCheck(
    logs: str,
    adapter: str | None,
    files: int,
    runs: int,
    successful: int,
    components: dict[str, int],
    decisions: list[DecisionCheck],
    findings: list[Finding],
)

The diagnosis of a log folder (see check_logs): ok when nothing would stop onboarding; summary(), to_dict(), to_json().

Attributes:

Name	Type	Description
`ok`	`bool`	False when a finding of level `error` would stop an onboarding.
`findings`	`list[Finding]`	what is wrong, where and how to fix it, each with a level (`error`, `warning`, `info`).
`decisions`	`list[DecisionCheck]`	one `DecisionCheck` per decision the adapter builds candidates and labels for.
`components`	`dict[str, int]`	the LLM-calling components found, with their call counts.
`files`	`int`	how many files were read, runs found and runs that succeeded.
`runs`	`int`	how many files were read, runs found and runs that succeeded.
`successful`	`int`	how many files were read, runs found and runs that succeeded.

Examples:

import tabagent

check = tabagent.check_logs("./agent_logs", decision="ToolShortlist")
print(check.summary())
assert check.ok  # the CLI exits with status 1 when this is False

ok `property` ¶

ok: bool

Whether no finding would stop an onboarding.

adapters ¶

Trajectory adapters: read an agent's native logs into TabSchema trajectories and decision points.

An adapter implements load(path) -> list[Trajectory], decision_points(traj, target), candidates(dp), labels(dp) and components(trajs), and exposes outcome_keys. A decision point's prefix holds only what was known before the decision ({<task fields>, "steps": [{"name": ..., ...}]}); labels may read the full run (for tool shortlisting, the tools the successful run used).

For tabagent.scan an adapter may also implement llm_calls(traj) -> list[LLMCall]: every LLM call of a run, with what it chose, what it was offered, the free text it wrote and the tokens, latency and cost the log records (read_usage reads the common shapes), plus scan_notes on how its logs record them. All the adapters below implement it.

OpenAIMessagesAdapter: chat-message runs (OpenAI, Anthropic Messages API format, LangChain dumps);
Tau2Adapter: τ²-bench results files;
CugaAdapter: CUGA agent trajectories (the API shortlister built in);
GenericJsonAdapter: any log with named steps, with your own candidates and labels;
detect_adapter(path): picks one of the above from the first files.

Every adapter reads files through tabagent.adapters.logfiles: encodings, damaged JSON and JSON Lines, linked folders, values nested too deeply, and large inputs read again from their files instead of held in memory. Whatever is skipped or repaired is reported in the adapter's warnings, runs of another log format included (detect_adapter(..., on_other_formats="error") refuses them with MixedFormatsError). tabagent.check_logs (CLI tabagent check-logs) diagnoses a log folder before an onboarding.

At decision time, live_prefix(adapter, run, target) builds a live run's prefix exactly as in training, and describe_adapter / adapter_from_description record and rebuild the adapter a head was trained with.

OpenAIMessagesAdapter ¶

OpenAIMessagesAdapter(
    granularity: str = "task",
    tools: Any = None,
    success: str | Callable[[dict], bool] | None = None,
    task_id_key: str | None = None,
    task_keys: Iterable[str] = TASK_KEYS,
    outcome_keys: Iterable[str] = OUTCOME_KEYS,
    max_chars: int | None = 20000,
)

Chat-message runs -> trajectories and tool-selection decision points (see the module docstring).

Candidates come from the run's own tools, else the tools catalog given here, else the union of the tools called across the loaded runs (argument hints then list the argument names observed in calls; calls answered with a tool-not-found error are left out). success is an outcome key or a function of the run; runs without an outcome count as successful, and runs whose outcome cannot be read (see tabagent.tabschema.trajectory.read_outcome) as failed; both are reported in warnings, with every file, line or record load skipped or repaired.

warnings `property` ¶

warnings: list[str]

Data notes for the onboarding report: files and records skipped or repaired, runs without an outcome or with an unreadable one.

load ¶

load(path: str | Path, *, lazy: bool | None = None) -> list[Trajectory]

The runs in one file or every .json/.jsonl file below a directory (see the module docstring); what was skipped or repaired, runs of another log format (τ²-bench results, step logs) included, is reported in warnings. lazy: read runs again from their files when needed instead of holding them (default: from 256 MB of logs, logfiles.LAZY_INPUT_BYTES).

decision_points ¶

decision_points(traj: Trajectory, target: str) -> list[DecisionPoint]

The decision points of a run (see the module docstring). The prefixes of one run share their step objects (see DecisionPoint): treat them as read-only.

labels ¶

labels(dp: DecisionPoint) -> set[str]

The tools chosen at the decision (for ToolShortlist: every tool the run called).

components ¶

components(trajs: list[Trajectory]) -> dict[str, int]

The decision this adapter builds -> number of successful runs with at least one tool call.

llm_calls ¶

llm_calls(traj: Trajectory) -> list[LLMCall]

The agent's LLM calls in a run, for tabagent.scan: one per assistant turn (see assistant_turns). A turn that calls tools is a NextToolCall: its choices are the tools it called, its catalog the run's tools (else the catalog given here), its free text the content and the arguments it wrote. A turn that only writes text is a TextReply. Usage is read from the turn's own fields (usage, usage_metadata, response_metadata, cost, ...) when recorded.

live_prefix ¶

live_prefix(run: dict, target: str | None = None) -> dict[str, Any]

The prefix of a live decision, built exactly as in training.

run has the shape of a logged run: messages holds the conversation so far (for ToolShortlist, the messages before the first tool call) and the run-level task fields (instruction, domain, ... as listed in task_keys) sit next to it, as in the logs the head was trained on. A field the training runs had and run lacks is missing from the prefix too.

Tau2Adapter ¶

Tau2Adapter(
    granularity: str = "task",
    tools: Any = None,
    include_policy: bool = False,
    max_chars: int | None = 20000,
)

Bases: OpenAIMessagesAdapter

τ²-bench simulations -> trajectories and tool-selection decision points (see the module docstring).

load ¶

load(path: str | Path, *, lazy: bool | None = None) -> list[Trajectory]

The simulations of one results file or of every results file below a directory. Files and records that are not τ²-bench results (another log format included) are skipped with a warning, and so are simulations that are not objects; simulations without a task id, naming a task the file does not list, or seen in an earlier file are reported in warnings. Runs are always held in memory (they are built from a results file and its task list), whatever lazy says.

CugaAdapter ¶

CugaAdapter(
    catalog: Any = None,
    success: Callable[[dict], bool] | None = None,
    outcome_keys: Iterable[str] = ("score", "eval", "actions_count"),
    outcome_steps: Iterable[str] = ("EvaluationResult",),
    candidates_fn: Callable[[DecisionPoint], list[Candidate]] | None = None,
    labels_fn: Callable[[DecisionPoint, dict], set[str]] | None = None,
    task_keys: Iterable[str] | None = TASK_KEYS,
    app_labels: str = "used",
)

Bases: GenericJsonAdapter

CUGA trajectories -> decision points for any agent node; candidates and labels built in for the ShortlisterAgent and the TaskAnalyzerAgent (see the module docstring). A run succeeds when score > 0; a missing or unreadable score counts as a failure and is reported in warnings. app_labels is "used" (the apps the run called) or "offered" (the apps the analyzer chose).

load ¶

load(path: str | Path, *, lazy: bool | None = None) -> list[Trajectory]

Every trajectory under path (recursive; results.json/metadata.json skipped); see GenericJsonAdapter.load. A run is an object with a steps list.

llm_calls ¶

llm_calls(traj: Trajectory) -> list[LLMCall]

The LLM calls of a run, for tabagent.scan: each step of an agent node (or with recorded prompts), one call per recorded generation (see the module docstring).

decision_points ¶

decision_points(traj: Trajectory, target: str) -> list[DecisionPoint]

See GenericJsonAdapter.decision_points. For the built-in decisions the labels are read from the full run now and kept (a few names per decision point), so they need not read the run again.

labels ¶

labels(dp: DecisionPoint) -> set[str]

Shortlister: the APIs the CodeAgent called after this shortlisting and before the next one. App analyzer: the apps the run called after it (app_labels="used"), or the apps it chose ("offered"). Read from the full trajectory.

GenericJsonAdapter ¶

GenericJsonAdapter(
    steps_key: str = "steps",
    step_name_key: str = "name",
    success: str | Callable[[dict], bool] | None = None,
    group_key: str | None = "task_id",
    outcome_keys: Iterable[str] = ("score", "eval", "evaluation", "reward", "success", "outcome"),
    outcome_steps: Iterable[str] = ("EvaluationResult",),
    candidates_fn: Callable[[DecisionPoint], list[Candidate]] | None = None,
    labels_fn: Callable[[DecisionPoint, dict], set[str]] | None = None,
    task_keys: Iterable[str] | None = TASK_KEYS,
)

Reads JSON trajectories with named steps, {..., "steps": [{"name": <component>, ...}, ...], <outcome>}: one per .json file, a list of them in one file, or one per line of a .jsonl file.

A run succeeds by its first outcome field among success, reward and score (see read_outcome); runs without one count as successful and are reported in warnings, and so are unreadable outcomes, which count as failures. Pass success (a field name or a function of the run) to decide otherwise. Prefixes keep only the top-level fields in task_keys (None keeps every field but the outcome keys); the fields left out are reported in warnings. The task id (group_key) is never in a prefix: it is the cross-validation group, not a feature.

Candidates and labels are benchmark-specific; pass candidates_fn(dp) and labels_fn(dp) (for example, the catalog of tools and the tools the successful run actually called), or use a dedicated adapter (tabagent.adapters).

load reads the files as described in tabagent.adapters.logfiles (encodings, damaged JSON, JSON Lines, linked folders) and reports in warnings every file, line or record it skipped or repaired; it refuses a folder that also holds runs of another log format. A step that is not an object, or has no string name, stays in the prefixes but is never a decision.

warnings `property` ¶

warnings: list[str]

Data notes for the onboarding report: files and records skipped or repaired, runs without an outcome or with an unreadable one, fields kept out of prefixes.

load ¶

load(path: str | Path, *, lazy: bool | None = None) -> list[Trajectory]

Trajectories from one file or every .json/.jsonl file below a directory (recursive, linked folders followed; hidden files and the results.json/metadata.json run summaries skipped). A .json file holds one run or a list of runs, a .jsonl file one run per line. A trajectory's id is its path below the directory (:<index> for a run in a list or on a line). Files, lines and records that cannot be read, records without the steps key and runs of another log format are skipped and reported in warnings (see on_other_formats). lazy: read runs again from their files when needed instead of holding them (default: from 256 MB of logs, logfiles.LAZY_INPUT_BYTES).

decision_points ¶

decision_points(traj: Trajectory, target: str) -> list[DecisionPoint]

Every step named target, with its redacted prefix (see redact; the prefixes of one run share their steps, see DecisionPoint). Steps that are not objects, or have no string name, are kept in prefixes but are never decisions.

live_prefix ¶

live_prefix(raw: dict) -> dict[str, Any]

The prefix of a live decision: the trajectory so far, redacted exactly as in training.

components ¶

components(trajs: list[Trajectory]) -> dict[str, int]

Step name -> number of successful trajectories containing it (the replaceable components), most frequent first (ties by name).

candidates ¶

candidates(dp: DecisionPoint) -> list[Candidate]

The legal candidates of a decision point (from candidates_fn).

labels ¶

labels(dp: DecisionPoint) -> set[str]

The chosen candidates, read from the full trajectory (e.g. the tools the successful run used). Available for decision points produced by this adapter's decision_points.

llm_calls ¶

llm_calls(traj: Trajectory) -> list[LLMCall]

The LLM calls of a run, for tabagent.scan: one per step that records an LLM call (a usage, model, prompt or messages field). When no step of the run records one, every named step counts as a call, marked inferred. The output is the step's output, data, result, response, completion, answer, decision or choice field (JSON text is parsed), else its other fields; usage is read with read_usage.

detect_adapter ¶

detect_adapter(path: str | Path, *, on_other_formats: str = 'skip', **kwargs: Any)

The adapter for the logs at path, chosen from the first files; kwargs go to its constructor.

The first records of the first files (up to 20, run summaries skipped) decide: CUGA trajectories over generic step logs, else the format most of them hold. Runs of another format are skipped by the adapter's load with a warning (on_other_formats="skip"), or stop it with MixedFormatsError ("error").

live_prefix ¶

live_prefix(adapter: Any, run: dict[str, Any], target: str | None = None) -> dict[str, Any]

The prefix adapter builds for a live run (the run so far, in the logged format) before the decision target.

Chat-message adapters take {<task fields>, "messages": [...], "tools": [...]}; CUGA and generic step adapters take the trajectory so far ({<task fields>, "steps": [...]}), redacted as in training.

Logs and adapters¶

scan ¶

ScanReport dataclass ¶

candidates property ¶

to_dict ¶

to_json ¶

summary ¶

to_markdown ¶

check_logs ¶

LogCheck dataclass ¶

ok property ¶

adapters ¶

OpenAIMessagesAdapter ¶

warnings property ¶

load ¶

decision_points ¶

labels ¶

components ¶

llm_calls ¶

live_prefix ¶

Tau2Adapter ¶

load ¶

CugaAdapter ¶

load ¶

llm_calls ¶

decision_points ¶

labels ¶

GenericJsonAdapter ¶

warnings property ¶

load ¶

decision_points ¶

live_prefix ¶

components ¶

candidates ¶

labels ¶

llm_calls ¶

detect_adapter ¶

live_prefix ¶

ScanReport `dataclass` ¶

candidates `property` ¶

LogCheck `dataclass` ¶

ok `property` ¶

warnings `property` ¶

warnings `property` ¶