Project

Podcast transcript

The episode Distilling LLM Agent Decisions into Cheap Classifiers, as text.

Distilling LLM Agent Decisions into Cheap Classifiers (22:51) is an audio overview of the paper, generated with NotebookLM from the paper and the code. The two voices are AI-generated; where they paraphrase or speculate, the paper is the authoritative source. Listen on the home page.

This transcript was made with speech recognition and lightly corrected (product and benchmark names); timestamps mark the start of each passage.

Where the conversation goes beyond the paper

• 1:30, "tens of thousands of tokens": the paper measures the replaced shortlisting call at roughly 21k tokens.
• 11:04, Recall@9: Recall@9 is the share of the tools the GPT-5.4 shortlister used on a successful run that appear in the top nine. It counts all of them, not a single "correct tool".
• 12:58, "failed exactly one task": the live result is Pass@1 averaged over up to three seeds per task, paired over 24 tasks. The difference, −2.1 points with a 95% CI of [−18.8, +16.7], is not a count of tasks.
• 13:56, "90-plus percent reliability": this is the hosts' gloss, not a paper figure. The paper reports task success on par with the GPT-5.4 shortlister, within that interval.
• 22:22, the closing speculation about frontier models that exist only to bootstrap classifiers is the hosts' own; the paper makes no such claim and keeps the frontier model in the loop for novel cases.

0:00 So, imagine your LLM agent needs to pick which tools to use for a specific task. Right. And right now, calling a frontier model like GPT-5.4 to make that choice, it takes about 7.5 seconds and costs around 5.2 cents. Yeah, which adds up fast. Exactly. But what if there was a classifier that makes the exact same choice in 2.682 milliseconds for a fraction of a millionth of a dollar? Oh, wow. We are literally talking about two tenths of a microcent.

0:31 That is a thousand X reduction in latency and a 100,000 X reduction in cost. It's crazy. So, how is that even mathematically possible? And, you know, as developers, what is the catch here? I know. It sounds like a typo in a benchmark report, right? Completely. But it is real. The mechanics behind it are outlined in this paper we're diving into today called TabAgent, distilling LLM agent components into cheap classifiers. It's by Levy and colleagues from IBM and the Weizmann Institute of Science. It's a great paper.

1:01 It really is. And the core mission of this deep dive is to unpack this research because it's forcing us to completely rethink where and how we deploy our most expensive compute when we're building agentic systems. We're going to break down what TabAgent is, why it matters, and how you can actually integrate it into your agents today. Perfect. Because, you know, to understand how TabAgent achieves this massive cost reduction, we first have to look at what LLMs are actually wasting their time doing inside these agentic loops. Exactly.

1:30 If you build agents, you know they are constantly making these closed set decisions again and again. Over and over. Right. The agent gets a user query, it enters a reasoning loop, and then it has to like shortlist tools from a catalog, or it has to pick a specific skill, or route a query to a sub-agent, or decide whether to activate a safeguard. And the defining characteristic of every single one of those examples is that the answer always comes from a fixed list of candidates. Yes. The agent is not writing a poem here, you know. It is literally picking item C from the list of A-Z. But right now, today, every single one of these multiple choice decisions is being routed through a full multi-billion parameter Frontier LLM call.

2:13 Yeah, it's wild when you think about it. It is. The agent is serializing tens of thousands of tokens. The system prompt, the user history, the JSON descriptions of every single tool in the catalog, all just to pick an API from a list. And that context window serialization is what completely dominates the agent's latency and cost on the critical path. But wait, hold on. If these are just multiple choice questions from a fixed catalog, why are we using generative models for this in the first place? I mean, isn't this exactly what traditional classifiers were built for?

2:44 That is the perfect question. Okay. And the short answer is zero-shot reasoning. Okay, right. We use generative models because they're incredibly good at entering a brand new situation, reading the really complex state of an agent, like its messy internal monologue, its past errors, its current goals, and then deducing which tool makes sense without any prior training. Which traditional classifiers struggle with? Exactly. Traditional classifiers, historically anyway, just couldn't parse that kind of highly unstructured dynamic text state. They need clean columns and rows.

3:18 Right, they need structure. Yeah. But the authors of this paper realize we've basically been looking at the problem backward. We do not need a frontier model to reason from scratch every single time, especially if it's a path the agent has already successfully walked before. Because the real decision data already exists, right? It lives in your agent's own successful past runs. Yes, exactly. If you have an agent deployed in the wild and it is successfully solving tasks, it has already done the heavy lifting of reasoning out which tools to use in which contexts. So the insight here, and this is brilliant, is to turn every one of those past decisions into a simple tabular data set.

3:56 Okay, so how does that look? Well, you create one row per decision candidate pair. So if your agent was faced with a specific context and it picked, say, tool A in a successful run, you label that row with a 1. Makes sense. And if it ignored tool B, you label that row with a 0. You're essentially capturing the frontier model's successful reasoning and mapping it to a super simple binary classification task. Oh, wow. And then you just train a small, cheap classifier on this table.

4:28 That's it. So you only change the decision component of the agent, the rest of the architecture, like the planner, the execution environment, the memory modules, all of that stays exactly as it is. You don't touch any of it. It really just boils down to this one idea. Many decisions inside an LLM agent are just multiple choice questions. And you don't need a frontier model to answer a multiple choice question that your agent has already answered correctly hundreds of times. That is the one sentence thesis of this whole paper. But, you know, having the theory to build a table is one thing.

5:00 Right. The execution is the hard part. Always. The engineering hurdle here is figuring out how you actually extract live execution traces, like the messy JSON logs, the free text thoughts, the varying system prompts, into a clean tabular format and doing it without losing the nuance of the context. Because if you just grab random keywords, the classifier is basically guessing blindly. Right. Exactly. So the paper solves this context loss problem with a specific three-step architecture. Okay. Let's break that down for the developers listening. What's step one?

5:34 Step one is called TabSchema. And it's actually an agentic system in itself. Oh, wait. Really? Yeah. Its only job is to figure out what data actually matters. It reads the successful trajectories and proposes features that might be relevant to the decision. Okay. So it looks at the logs and might hypothesize, for example, that the length of the user's prompt is a crucial signal. Exactly. Or it might flag specific keywords in the agent's thought process or the success state of the previous API call. It doesn't just list these ideas, right?

6:05 No. And this is the cool part. TabSchema actually writes Python code to extract those specific features from the raw traces. Oh, that's clever. Right. It tests that Python code in a sandbox to ensure it executes without errors over all the trajectory data. So it's like a team of analysts studying your best employees past decisions. That's a great way to put it. Yeah. Like they aren't making the choices themselves. They are literally writing down exactly what metrics the employee looked at before making a choice. And then they're building a rubric. Exactly.

6:35 But wait. They don't just keep every single metric they find, do they? Because some of those extracted features are going to be completely useless noise. They absolutely are. So to filter out that noise, TabSchema runs the extracted features through a panel of LLM judge models. Oh, OK. Yeah. They use something called a meta judge. The panel evaluates whether each feature genuinely correlates with the successful decision. So they throw out the weak signals. Exactly. They keep only the features that actually matter. And the final vetted output of this entire process is a feature card. A feature card.

7:08 Got it. Basically, it's a precise schema defining exactly how to turn your messy logs into structured columns. OK. So you have this feature card, which is brilliant for structuring the data you have. But what happens if your agent hasn't encountered enough edge cases to train a robust classifier? Because in the real world, trajectory data for specific tool combinations can be incredibly thin. And that brings us to step two, TabSynth. TabSynth. Yeah. This is designed to solve the exact data scarcity problem you just mentioned. But without introducing the fatal flaw of most synthetic data generation.

7:43 Which is hallucination. Exactly. Hallucination ruins data sets. So TabSynth is a deterministic rule-based paraphraser. It rewrites the free text parts of a training row, like the agent's internal thoughts or the status summaries of previous actions. To put that in ML terms for our listeners, it's basically a highly constrained data augmentation technique. Yeah, that's exactly what it is. It's like a highly disciplined copy editor who gives you variations of your notes so you have more examples to train on. But they are strictly forbidden from inventing new facts.

8:15 Yes. It copies the candidate tool, the label, and all the extracted tabular features verbatim. So if the agent successfully used a calculator tool to multiply three and four, TabSynth might rephrase the agent's thought from, I need to multiply to calculating the product. But it will never alter the fact that the calculator was the correct choice. Never. It never invents a new positive example from scratch. And that strict label preservation is why it works so well. Okay, so once you have your feature card from step one and your augmented data set from step two, you reach step three, which is TabHead.

8:52 This is the actual classifier that makes the routing decisions in production. Okay, and what kind of model is it? In the paper, they use a gradient boosted decision tree, specifically CatBoost. CatBoost, right. It takes the feature card, processes the current state of the agent, and scores every candidate in the catalog in the single pass. But wait, why CatBoost? If we are trying to replicate the performance of a massive neural network like GPT-5.4, why fall back to gradient boosted trees instead of training, say, a small neural network classifier? It's because tabular data generated by agents is incredibly heterogeneous.

9:30 Oh, right. Because of the different data types. Yeah, you have categorical features, you have Boolean flags about whether an API failed previously, and then you have numerical features like prompt length. And neural networks hate that. They do. Deep neural networks often really struggle to learn from this kind of unscaled mixed type tabular data without massive amounts of tuning. They tend to just overfit. Yeah, I've seen that happen a lot. But gradient boosted trees, by their mathematical nature, handle heterogeneous features and missing values exceptionally well right out of the box.

10:05 They split on the features naturally and require far less data to actually converge. That makes perfect sense. And there is a crucial safety mechanism built into this CatBoost model, right? It includes a confidence gate. So it doesn't just output a prediction, it outputs how sure it is about that prediction. Exactly. So if the classifier encounters a state that is wildly different from its training data and its confidence score drops below a certain threshold, it simply hands the routing decision back to the original frontier LLM. It essentially knows what it doesn't know. Right, which is huge for production reliability. But a clever architecture on paper is meaningless if it degrades the agents and end performance and practice.

10:43 Oh, absolutely. So the benchmark results are where these claims of 100,000x cost reduction really meet reality. Let's get into the numbers. Let's look at AppWorld first. It's a notoriously complex benchmark designed to test an agent's ability to shortlist tools across a bunch of different applications. And the results there are pretty staggering. On AppWorld, TabAgent recovers essentially all the tools the GPT-5.4 shortlister used on its successful runs. Yeah, they measure it using recall at 9, and TabAgent hits between 92 and 100% across the different applications.

11:17 And just for context for everyone listening, recall at 9 means that the definitively correct tool was present in the top nine candidates the classifier ranked. Which is critical because agents usually pull a shortlist of tools into their context window, not just a single one. Exactly. And the crazy part is, it achieves that 92 to 100% recall while operating at that micro-cent cost level. More than a thousand times lower latency and a hundred thousand times lower cost per decision. And the efficiency actually scales as the problem gets harder. Oh, really?

11:48 Yeah. Take SkillsBench, for example, where the agent has to choose the right action among a massive catalog of 195 different skills. That's a huge catalog. It is. And TabAgent ranks above every single retrieval-based method and every LLM baseline they tested. See, the dynamic there is fascinating. Because as the catalog of skills grows, the LLMs start drowning in their own context window. Oh, big time. They lose track of the nuances buried in thousands of tokens of tool descriptions. But the CatBoost classifier doesn't have a context window problem at all.

12:21 Nope. It just crunches the extracted tabular features for all 195 candidates in milliseconds. Exactly. Now, to prove this isn't just some artifact of offline benchmarking, the authors did something really cool. They hot-swapped TabHead into a running CUGA agent on 24 held-out AppWorld tasks for a live test. A live test. Okay, what happened? They measured the end-to-end task success rate of the entire system. And with TabHead making the routing decisions, the task success rate was 27.1%. Okay.

12:52 With the original GPT-5.4 shortlister doing the routing, the success rate was 29.2%. Okay, wait, let me push back on that and do the math for a second, because it's important. 27.1% versus 29.2%. That's a 2.1% absolute drop. Right. But on a sample size of exactly 24 tasks, that literally means the TabAgent setup failed exactly one task that the full LLM setup passed. Yes, just one. So let's be real.

13:21 On a sample that small, the confidence interval of the difference includes zero. We completely lack the statistical power to prove the performance is definitively worse. On par in this case literally just means we don't have enough data to prove a negative. I mean, that is just the harsh reality of evaluating live complex agents. The runs are expensive, so the sample sizes stay small. Yeah, true. The authors are very transparent about the statistical limits there. But the engineering trade-off is clear. You are retaining the vast majority of your end-to-end system reliability, while cutting the compute cost of your decision-making by five orders of magnitude.

13:56 Yeah, for most production environments, retaining 90-plus percent reliability for a fraction of a microcent is a trade-off teams will make instantly. Absolutely. Which leads to the finding that actually surprised the authors the most. Oh, the ablation studies. Yeah, they ran ablation studies to figure out where the actual intelligence of TabAgent lives. And it turns out the massive performance gains do not come from using CatBoost. Wait, really? Yeah, the magic is entirely in the representation.

14:26 The feature card that was generated by TabSchema. Ah, so it's the data structure, not the model. Exactly. They tested everything from simple, unboosted decision trees to complex, deep tabular foundation models. And as long as the models were fed the structured data from the feature card, they all performed exceptionally well. Oh, wow. But if you take the feature card away and try to feed those same models, just the raw text or basic text embeddings of the logs, their performance completely collapses. That is wild.

14:56 The intelligence is completely in the automated feature extraction, not the classifier weights. Bingo. But wait, if the feature extraction is doing all the heavy lifting, what happens when it encounters a completely new tool or environment it has never extracted features for before? Because look, we are claiming astronomical cost savings here. So if we don't look at the honest limits, people will rightly write this off as a sales pitch. Right. And we have to be brutally honest about where it breaks down. So what are the failure modes? Well, the most glaring limitation is the cold start problem.

15:28 Right. Because the feature extraction relies on analyzing successful past logs, TabAgent absolutely needs successful trajectories to exist first. It cannot handle a brand new domain zero shot. Meaning if your agent has never successfully solved a task in a specific environment, there are no logs for TabSchema to analyze and therefore no feature card can be built. Exactly. You have to run the expensive frontier model to generate the initial data. You can't skip that step. Makes sense. And because it learns exclusively from that source agent's past logs, it learns that specific agent's decision boundaries.

16:02 So by design, it inherits the source agent's biases. Yeah, it's a clone. Right. If your frontier LLM had a weird habit of favoring one specific search tool over another, when both work equally well, TabAgent will happily clone that weird habit right into the classifier. It's a mirror, not an independent reasoner. And that mirroring effect creates what we call routing ceilings. Routing ceilings. Yeah. On policy gating and routing decisions, TabAgent is built to match a strong fixed default. Like the frontier model's tendency to pick the universally best sub-agent.

16:35 Okay. It executes that default highly efficiently, but it doesn't possess emergent reasoning to outsmart the baseline heuristic. If the source model was suboptimal, TabAgent would just be really efficiently suboptimal. Okay. That makes total sense. I also want to dig into TabSynth for a second, the paraphraser, because the paper notes a very specific failure mode there. Oh, the Spotify benchmark. Yes, on the Spotify benchmark. Using TabSynth to augment the data actually hurt performance. It did.

17:03 The R-precision, which measures the proportion of retrieved tools that are actually relevant, dropped from 70% down to 61% when they use the paraphrased data. Yeah, it was a significant drop. So, wait, if TabSynth is strictly label preserving and just rephrasing things, why would it actually damage the model's accuracy on Spotify? It all comes down to the lexical nature of that specific domain. Lexical meaning the exact wording. Exactly. On a platform like Spotify, the exact wording of song titles, artist names, or specific API parameters are incredibly strong, necessary signals.

17:40 Oh, I see where this is going. Yeah. So, when TabSynth introduces paraphrased diversity, it tries to vary the language in the agent's thoughts. It might paraphrase, you know, play the album to execute the collection of songs. Right. And in doing so, it dilutes the clear lexical signals with synonyms that aren't actually helpful. It might even break the keyword overlap the classifier relies on to match the thought to the tool description. It adds noise exactly where precise wording was the most important feature. Precisely.

18:10 Paraphrasing bohemian rhapsody into the audio track regarding bohemian rhapsody just completely confuses the classifier. Exactly. It breaks the literal string match. That makes perfect sense. And the final limitation, which is actually a safety feature, is that the frontier model does not go away. No, it doesn't kill the LLM entirely. Right. TabAgent just acts as a highly efficient filter on the critical path. The frontier model stays deployed in the background as a fallback for those rare, novel edge cases triggered by that confidence gate we talked about.

18:43 Right. So, if you are building an agentic system and you are comfortable with those constraints, the immediate question is the friction of adoption. How much data and compute do you actually need to gather to make this pipeline work? Exactly. Because everyone in ML says low data. Every paper claims their method is low data. But what does that practically mean here? Am I going to have to burn $1,000 of API credits just to generate the successful logs needed to train the cheap classifier? That's the best part.

19:12 No. The data requirement is remarkably low. Really? Yeah, onboarding TabSchema is a one-time step per agent and per catalog. It does not require thousands of runs. It literally only needs tens to low hundreds of successful trajectories. Wow. Okay. To give you a number. On the highly complex AppWorld benchmark, the CatBoost head reaches 90% of its full quality from just 118 successful runs. 118 runs? That's nothing.

19:41 Generating 118 successful runs with the GPT-4 or Claude is a rounding error in your cloud budget compared to running those models in production on every single user request perpetually. It really is. And the maintenance overhead is minimal too. What happens when I update my agent? Well, say you update your agent's system prompt to adjust its persona. But the overall JSON log format stays the same. You do not have to rebuild the feature card or run TabSchema again. Oh, nice.

20:11 You only have to refit TabHead on the new logs. Because it's a CatBoost model operating on a really small tabular dataset, retraining takes about 45 seconds on a standard CPU. That's incredible. Yeah, you don't even need to spin up a GPU to maintain it. So since the friction is that low, let's talk about how you, the listener, can literally implement this into your agent code base today because the code is entirely open source under the Apache 2.0 license. It's all out there. You can literally open your terminal right now and run pip install tabagent. And depending on your current architecture, there are three practical ways to integrate it.

20:46 Okay. What's the first way? First, if you already have your agents logs neatly formatted into a tabular structure, you don't even need the full pipeline. You can just use their scikit-learn estimators like TabAgentRanker to train a ranker directly on your existing table. Nice and easy. Second, if you have raw logs and you want the whole automated pipeline we just talked about, you can point the CLI directly at your agent's traces. You just run tabagent onboard and tell it which specific decision node you want to optimize, like tool shortlist.

21:16 And it will run TabSchema, build the feature card, and train the classifier from scratch. Exactly. And third, if you happen to be developing on standard open source benchmarks like SkillsBench, you can actually skip training entirely. Oh, they have pre-trained models. They do. You can download a pre-trained TabHead directly from Hugging Face and plug it into your evaluation pipeline right away. That's fantastic. All the code, the detailed documentation, and the links to those pre-trained heads are available at github.com/dolev31/TabAgent. It's definitely worth checking out.

21:48 Bringing it all back to the core thesis one last time. Many decisions inside an LLM agent are simply multiple choice questions. And you do not need a frontier model to answer a multiple choice question that your agent has already answered correctly hundreds of times. That's so true. It really forces a profound shift in how we architect these systems. It does. We are moving away from monolithic brute force LLM calls for every tiny step in the loop toward a more modular, cost-effective design that fundamentally respects the limits of our compute budgets.

22:21 Absolutely. And it leads me with this thought. If agents can automatically use a system like TabSchema to distill their own decision boundaries from their own successful logs, it really makes you wonder about the future. Yeah, so. Well, are we moving toward a future where massive frontier models only exist temporarily? Like spun up just long enough to bootstrap swarms of hyper-efficient, self-improving microclassifiers before being spun right back down? That's a wild thought. Something to mull over the next time you watch your API bill climbing.