Blogpost & Paper Reviews

“As AI agents are becoming widely adopted in the financial services industry” - what sub-industries are typically included in the financial services category?

Background knowledge: Financial services usually covers banking (retail, commercial and investment), insurance, asset and wealth management, capital markets (brokerage, exchanges, trading), payments and fintech, and lending such as mortgages and consumer credit. Accounting, audit and tax firms are normally classed as professional services, even though they serve financial firms.

Inference: That distinction weakens the paper's opening statistic. The 71% adoption figure comes from a survey of financial services firms, which is a different population from the CPA firms whose work FAB tests. It shows that auditors' clients are adopting AI, not that auditors are.

Why do audits happen in the first place? Who wants them to happen and who is paying for them? What is the incentive structure?

Background knowledge: An audit gives outsiders independent assurance that a company's financial statements are fairly stated.

• Who requires it: US public companies must be audited under securities law. Private companies are usually audited because a lender, investor, buyer, regulator or grant-maker requires it, often as a loan condition.
• Who pays: the audited company. That is a conflict of interest, controlled by independence rules, hiring through the board's audit committee at public companies, outside inspection (the PCAOB for public company audits, peer review for private ones), and the auditor's legal liability and reputation.
• Incentives: management wants a clean opinion so it can borrow or raise money on better terms; investors and lenders want numbers they can rely on; the auditor wants the fee but bears the cost if a failed audit comes to light.

Inference: The blog's sample company, a synthetic manufacturer with a bank term loan that carries a loan-to-collateral covenant, fits the typical lender-driven private audit.

Why did Modus publish this benchmark in the first place? What do they get out of it?

Inference: The benchmark serves that business in four ways: credibility on audit quality with partner firms, clients and regulators; recruiting AI researchers; an internal way to choose models (the blog argues for model-agnostic infrastructure); and influence over how the profession measures AI audit quality. Note the incentive: a company that buys audit firms and supplies their oversight benefits from the finding that agents need professional oversight.

“this domain is highly suitable for the strong information retrieval and reasoning capabilities that agents have shown to possess”

Why?

Inference: The fit comes from the shape of the work. Evidence is mostly text and tables in PDFs and spreadsheets; procedures are written out step by step in templates; many steps are lookups, comparisons and arithmetic an agent can do in code; and most answers can be checked against documents.

From the paper (Table 3): The results show where the fit ends. Average strict pass is 94.5% on journal-entry completeness, which is mostly mechanical, but 7.6% on revenue and 4.2% on accounts receivable, where the agent must judge which evidence to trust. "Highly suitable" holds for retrieval and calculation, not yet for evidence judgment.

“further progress in the evaluation and training of frontier models is increasingly constrained by the amount of accessible data”

What do they mean by "training" here?

From the paper: They mean training frontier models. The conclusion says the framework can generate tasks for model training and evaluation in privacy-sensitive domains, and the citations (Villalobos et al.; Muennighoff et al.) are about running out of human-written data to train large models.

Inference: The most likely use is post-training. Each generated binder plus its rubric is a reinforcement learning environment with a checkable reward, and expert or model trajectories can serve as fine-tuning data. The generator can make unlimited consistent binders, which real client data can never supply. That also points to a commercial angle: selling training environments to AI labs in a domain where real data can't be shared.

“use large language models (LLMs) to propose and validate tasks”

What does this mean?

Background knowledge: In those methods an LLM explores an environment such as a website and writes candidate tasks, for example "find the cheapest flight from this page." An LLM then attempts each task and checks whether it is feasible and was solved, and the tasks that pass are kept. No human writes the tasks.

Inference: The contrast the paper draws is that LLM-invented tasks reflect what a model finds plausible, and an LLM checking an LLM can share its blind spots. FAB grounds its values in private historical statistics and uses auditors as reviewers. It still uses LLMs at many steps: proposing and ranking variables, extracting values from historical binders, adding graph edges, writing feasibility tests, and copying rubric structure across tasks.

“require many hours of work by domain experts”

What role do the domain experts play exactly?

From the paper: Auditors contributed a little over 1,100 hours in four kinds of work:

• Design: fifteen auditors identified the information used in these audits; three auditors assessed LLM-proposed variables, with a senior auditor (more than ten years of experience) consolidating; auditors defined the graph's variables, dependencies and accounting rules.
• Quality control: auditors reviewed each round of generated binders for inconsistencies, implausible content, formatting problems and missing documents.
• Labeling: at least one CPA completed or reviewed each reference workpaper and a senior auditor adjudicated; a senior auditor wrote example rubrics, and CPAs revised all rubrics against model outputs.
• Analysis: auditors ran the failure review and estimated manual completion times.

Inference: In the expert-built benchmarks the paper contrasts itself with, experts write each task by hand. Here they mostly write rules and check outputs, which is what lets the generator scale.

“Task structure in FinancialAuditBench. The agent uses the provided supporting documents to perform the appropriate audit procedures in a workpaper template”

What is a workpaper and what does it mean to complete one?

From the paper: A workpaper is a spreadsheet, built from a predefined template, where auditors record the procedures performed, the evidence examined and the conclusions reached. Figure 1 shows a cash lead sheet (a summary of one account area) listing each bank account's balance per the trial balance (the list of all ledger account balances), per the bank reconciliation, and whether the two agree.

From the blog post: The cash template lists 31 procedures across 7 tabs: lead sheet, bank confirmation, bank reconciliation, outstanding checks, cutoff, depository financial condition, and presentation and disclosure. For each procedure the preparer records applicability, execution, evidence, result, open issues and sign-off.

Inference: Completing one means performing each listed procedure with the binder's evidence, filling in the tables with amounts, dates and references to source files, recording exceptions, and writing conclusions the evidence supports. The workpaper is the record that the work was done and what it showed.

“a domain that examines whether a company’s reported financials are supported by sufficient evidence and are accurately reported within a set of tolerance bounds”

Is this “sufficient evidence” always internal information or could it include external knowledge?

Background knowledge: It includes external evidence, which is preferred. Auditing standards treat evidence as more reliable the more independent its source: evidence sent directly to the auditor by an outside party, such as a bank confirmation, is strongest; then outside documents held by the client, such as bank statements and vendor invoices; then documents the client creates itself. Evidence the auditor produces, such as recalculations or watching an inventory count, is also strong. Public information is used too, for example the FDIC insurance limit or a bank's credit rating.

From the paper: Binders contain client-provided records, third-party evidence, auditor-developed materials and public information. One of the four failure types is relying on client documents where third-party evidence was needed.

Inference: "Sufficient" refers to both quantity and quality: enough items, from sources reliable enough for the assessed risk.

“a domain that examines whether a company’s reported financials are supported by sufficient evidence and are accurately reported within a set of tolerance bounds”

What’s the purpose of the tolerance bounds? Shouldn’t accounting be deterministic?

Background knowledge: The tolerance bound is materiality: the size of misstatement that could change a reader's decisions. Bookkeeping arithmetic is deterministic, but financial statements are not, for two reasons. Many balances are estimates, such as the allowance for credit losses (expected bad debts) or obsolete inventory. And auditors test samples rather than every transaction, so they give reasonable assurance, not certainty; checking every cent would cost more than the extra accuracy is worth.

From the blog post: The sample engagement sets overall materiality at $64,880, performance materiality (a lower working threshold) at $48,660, and a clearly trivial threshold of $3,244. In the cash run, a $108.29 difference between two bank statements was trivial in amount but was still flagged, because contradictory documents raise a question about the evidence itself.

“Auditors identify areas where significant reporting errors could occur”

What does this mean?

Background knowledge: This is risk assessment. For each account, auditors judge how likely a material misstatement is for each assertion, meaning each claim the numbers make: the item exists, the record is complete, it is valued correctly, it is in the right period (cutoff), and it belongs to the company. Risk rises with size, complexity, reliance on estimates, and incentives to misstate. Revenue near year-end is a standard high-risk area, because sales can be pushed into the wrong year to meet targets.

From the blog post: The sample planning file rates cash existence as moderate risk and revenue occurrence as high.

From the paper: In FAB this step is already done; the agent receives the assessed risks as input.

“Obtain supporting evidence”

How? What are they looking for?

Question: How? What are they looking for?

Background knowledge: Auditors use a standard set of methods: inspecting documents (invoices, contracts, bank statements); external confirmation, where a bank or customer replies directly to the auditor; recalculation; reperforming a client process; analytical procedures, which compare figures with expectations; observation, such as attending an inventory count; and inquiry. In practice the auditor sends the client a list of "prepared by client" (PBC) requests and follows up on gaps. They are looking for evidence on each assertion, for example that a recorded sale happened, that all liabilities are recorded, and that balances are valued correctly and recorded in the right period.

From the paper: FAB skips the request-and-follow-up loop. The binder already contains the evidence, and agents can't ask for more.

“test the company’s reported amounts against that evidence”

What does this mean?

Background knowledge: It means comparing a recorded number with independent evidence and investigating any difference. The main techniques:

• Tie-out: agree a balance to the trial balance or a supporting schedule.
• Vouching: start from a recorded entry and go back to its source document, which tests whether it is real.
• Tracing: start from source documents and go forward to the ledger, which tests whether everything was recorded.
• Recalculation and confirmation.

A difference is an exception; if it matters, the auditor proposes an adjusting journal entry (AJE).

From the paper: Figure 1 shows payroll cash of $42,907 per the trial balance against $41,395 per the reconciliation, so the row is marked AJE. The text's example traces a recorded sale to the customer order, invoice and shipping confirmation.

My arithmetic: The Figure 1 difference is $1,512.

“Thus, a data room used for evaluation or training requires a set of financial records whose amounts, dates, and relationships are internally consistent across documents.”

Does it require just internal consistency or completeness/comprehensiveness as well?

From the paper: The stated requirement is consistency. Completeness is handled more loosely: auditors identified the information each workpaper needs, reviewers checked generated binders for missing documents, and binders provide "much of" the needed documentation. Some gaps and inconsistencies remain, and rubrics skip the parts they affect.

Inference: Both are required, at different levels. Consistency has to be close to total, because one disagreeing number across 179 files makes the answer key ambiguous. Completeness only has to cover the evaluated procedures, not every record a real company would have. One sign of this: Figure 4's example binder includes inventory and payroll templates, but neither is among the eight evaluated areas.

“provide priors for a subset of selected variables”

What do they mean by variables here, and why are we subsetting them?

From the paper: Variables are the quantities the graph uses to describe a company, such as annual revenue, the payroll-to-revenue ratio, or the share of raw material wasted in production. Private priors cover six financial ratios, sixteen features of financial records, and summary statistics for adjusting journal entries and accounts payable activity.

Inference: They subset for four reasons:

• Privacy budget (the total privacy loss allowed): each released statistic uses some of it, so more statistics means more noise in each or a weaker guarantee.
• Need: deterministic variables such as totals are computed, not sampled.
• Expert coverage: many values can be set from public data or auditor judgment. Private data matters most where, as the paper says, auditors could not confidently judge whether a value was realistic.
• Cost: an LLM had to extract each variable from every historical binder.

“a synthetic engagement binder”

What does this phrase mean?

From the paper: An engagement is the audit of one company for one reporting period. The binder is the set of materials for one engagement: planning documents, client-provided records, third-party evidence and blank workpaper templates. "Synthetic" means the framework generated it for a fictional company. Binders average 179 files.

From the blog post: The sample company is Quelvox Industrial Holdings, a fictional manufacturer, with a 190-file binder.

Background knowledge: "Binder" is a holdover from paper audit files. Today it is an electronic folder, as in Figure 4.

“single-attempt task pass rate of 69.17% (Pass@1)”

What is this metric? What are the numerator and the denominator? And how is passˆ8 different?

From the paper: Pass@1 is the probability that a single attempt strictly passes, meaning every applicable rubric check is met, averaged over the 45 completion tasks and 8 runs per task. Pass^8 is the share of tasks that pass on all 8 attempts.

My arithmetic: The numerator is strictly passing runs and the denominator is 45 × 8 = 360 runs. Opus's 69.17% is 249 of 360. Its pass^8 of 44.44% is 20 of 45 tasks.

Background knowledge: Pass^k comes from τ-bench. If a task passed c times out of n = 8 runs, pass^k for that task is usually estimated as C(c, k) / C(n, k), where C counts the ways to choose k items, then averaged over tasks. Pass@k, at least one success in k attempts, is 1 − C(n − c, k) / C(n, k).

Inference: The gap between the two is the point. An agent that is right two times in three on average is right every time on fewer than half the tasks.

“Overall, we believe the quality, scale, and complexity of FinancialAuditBench make it a realistic and representative benchmark for evaluating agents’ abilities to complete common financial auditing procedures.”

How do they gauge the quality of the benchmark itself?

From the paper: They don't measure it; "we believe" carries the claim. The evidence offered is process: 1,100 auditor hours, CPA-prepared reference answers reviewed by a senior auditor, rubrics revised against model outputs for fairness, and an auditor failure review. They state that they report no inter-annotator agreement (how often independent graders agree), give no human accuracy estimate, and call evaluating realism an open challenge.

Inference: The quality claim rests on who built the benchmark, not on measured properties. Measurements that would help: a blind test where auditors try to tell synthetic binders from real ones; agreement rates between independent CPA graders; agreement between the LLM judge and CPAs; a human pass rate on the same rubrics; and error rates for each rubric check.

“containing 90 tasks across six synthetic audit engagements”

What's the difference between a task and an engagement?

From the paper: An engagement is one synthetic company's audit; there are six, each with its own binder. A task is one workpaper in one engagement, either completed from a blank template or reviewed. Each binder has 7 or 8 workpapers, giving 45 completion tasks and 45 review tasks.

From the blog post: Accounts payable appears only in manufacturing binders (3 tasks); every other area has 6 tasks.

My arithmetic: 3 manufacturing binders × 8 + 3 staffing binders × 7 = 45, so there are three binders per industry.

“Benchmarks for AI systems. Benchmarks for agent evaluation have progressed from information retrieval tasks to longer-horizon, real-world tasks.”

In what ways specifically does the general literature on benchmarks for AI systems inform the work they've done here?

From the paper: The related-work section mostly positions FAB rather than crediting sources, but several methods come from that literature:

• Task packaging in the Harbor format, which bundles instructions, a container environment and a grading script.
• Reliability measured with pass^k from τ-bench.
• Automated task generation, as in SWE-smith and AgentSynth, which motivates a generator instead of hand-written tasks.
• Long-horizon knowledge-work benchmarks such as TheAgentCompany as the target task style.
• Spreadsheet grading, where FAB sits between exact cell coordinates (SpreadsheetBench) and LLM grading (MBABench).

Inference: Rubric grading also echoes the legal-reasoning rubric work from co-author Hockenmaier's group (Lee et al., 2026), and the privacy method builds on differentially private synthetic data work such as AIM and Private Evolution.

What are the shortcomings of the other accounting benchmarks they cited that necessitated their methodology?

From the paper: The paper describes these benchmarks rather than criticizing them. APEX–Accounting tests bookkeeping work (recording transactions, reconciling accounts, preparing reports), not audit. AuditFraudBench tests fraud judgment from public company filings and regulatory evidence. FinMaster's audit tasks are finding errors injected into transaction records. FinanceBench is question answering on public filings, and the CPA-exam studies test exam knowledge.

Inference: None of them gives an agent a full set of private-style client records that agree with each other and asks it to perform and document fieldwork. Public-data benchmarks can't include the private documents that fieldwork depends on (bank statements, invoices, confirmations), and hand-built ones don't scale. That combination is what led them to a generator with privacy-protected priors.

“a workpaper is a spreadsheet organized using a predefined template in which auditors document procedures performed, evidence examined, and conclusions reached”

In what way does the template organize this spreadsheet?

From the blog post: The cash template has a "Library Procedures" tab listing all 31 procedures. Each has dropdown fields for applicability, requirement, execution state, evidence state, result, issue state and sign-off, plus the procedure text, the assertions it tests and the tab where it is performed. Each numbered tab (<.1> to <.7>) covers one sub-area, with a header (client, year-end, materiality), a purpose statement and tables for the work. A Planning Context tab holds materiality and sample sizes.

From the paper: Each worksheet is one or more vertically stacked tables with fixed columns. Agents may add rows but may not change columns.

Inference: Fixed columns make the workpaper both a checklist for the preparer and a structure a program can grade.

“a workpaper is a spreadsheet organized using a predefined template in which auditors document procedures performed, evidence examined, and conclusions reached”

How can you document procedures performed and evidence examined in a single spreadsheet? Does that not require significant amounts of text that might not fit?

Background knowledge: Space isn't the constraint. An Excel cell holds up to 32,767 characters, and workpapers mostly hold numbers and short comments. Evidence is cited by reference (file names or evidence index codes) rather than copied in; the documents stay in the binder. Commercial audit software links each cell to its source document.

From the blog post: The cash run added an Evidence Index tab listing sources EV-01 to EV-20, and comment cells cite those codes.

From the paper: The binding constraint is reviewer attention. Auditors found that agents documented at length how they reached an answer while leaving the evidence and conclusion less clear, wrote in sentence fragments, and formatted cells poorly.

“including planning documents”

What do planning documents mean here?

From the paper: Documents that record decisions made before fieldwork. Figure 4 lists Engagement Scope and Case Facts, a Materiality Memo, an Assertion-Level Risk Assessment, an Audit Selection Plan and a Planned Procedure Register.

From the blog post: The sample binder's planning folder also has an accounting policy register, a controls and IT strategy, a significant and fraud risk register, a related party register, an engagement calendar, a procedure window register (the allowed date ranges for evidence), and an unadjusted differences register.

Inference: These stand in for the planning and risk-assessment phases a senior auditor would normally complete, so the agent starts fieldwork with the decisions a staff auditor would be handed.

“blank workpaper templates”

Why are there multiple templates?

From the paper: There is one template per audit area, because each area needs different procedures and evidence. The evaluated areas are accounts payable, accounts receivable, cash, debt, fixed assets, revenue, journal-entry completeness and journal-entry risk testing.

Background knowledge: Firms organize workpapers by account area in the same way, so different staff can work in parallel and reviewers can find each area.

Inference: Figure 4's example binder also shows inventory and payroll templates, which are not among the eight evaluated areas. Either binders include templates that aren't scored, or the figure reflects an earlier version.

“The benchmark includes financial documents for six synthetic companies spanning two industries, manufacturing and staffing services”

Is there a reason it focused on these two industries specifically?

From the paper: No reason is given.

Inference: Probably two reasons. First, they contrast well: manufacturing centers on inventory, fixed assets and shipment cutoff, while staffing centers on payroll and client billing, so together they cover goods and services. Second, both are common clients of mid-sized CPA firms like the ones Modus acquires, so the historical corpus probably had enough binders of each. That matters for differential privacy: the noise added to each statistic doesn't shrink with sample size, so statistics are only useful when many binders contribute.

“Each synthetic company’s binder includes either seven or eight workpapers, depending on industry, that can be independently completed, where a single task corresponds to completing or reviewing one workpaper.”

Why are there multiple workpapers, and what's the difference between them?

From the paper and blog post: Each workpaper covers one audit area and can be completed on its own. Manufacturing binders have eight; staffing binders have seven because accounts payable appears only in manufacturing. The blog maps areas to procedure types: tie-out and reconciliation, third-party confirmation, vouching and tracing, recalculation, cutoff, estimates and judgment, and presentation and disclosure.

Background knowledge: What each area mainly involves:

• Cash: confirm bank balances and test reconciliations.
• Accounts receivable: confirm customer balances and test the aging and the bad-debt allowance.
• Accounts payable: search for unrecorded liabilities.
• Debt: confirm loans, recompute interest and check covenants.
• Fixed assets: test additions and depreciation.
• Revenue: test that sales happened and are in the right period.
• Journal-entry completeness: prove the population of entries is complete.
• Journal-entry risk testing: select and test unusual entries, a fraud procedure.

“In aggregate, FinancialAuditBench comprises 45 main tasks that measure workpaper completion performance and 45 corresponding review tasks.”

What's the difference between a completion and a review task?

From the paper: A completion task gives the agent a blank template. A review task gives it a workpaper already completed by a randomly chosen model from the main experiments, plus the same supporting files, and asks it to find and correct errors while keeping correct work.

Background knowledge: This mirrors the two roles in a firm: a staff auditor prepares and a senior auditor reviews.

Inference: Review is harder than it sounds. The reviewer must decide what to trust, and changing correct work is penalized; the paper notes that some workpapers scored lower after review.

“Agents receive an audit plan and assessed risks as inputs”

What does "assessed risk" mean, and why is that being given as an agent's input?

Background knowledge: Assessed risk is the auditor's planning judgment of the risk of material misstatement for each account and assertion, for example revenue occurrence rated high and cash existence moderate. It sets the nature, timing and extent of testing: higher risk calls for stronger evidence and larger samples.

From the paper: It is an input because FAB evaluates only substantive fieldwork. Planning and risk assessment come earlier and decide what procedures are done, so the agent is handed those decisions, as a staff auditor would be.

Inference: Fixing the risk assessment also fixes the procedures and sample sizes, which keeps the answer key well defined. If each agent assessed risk itself, every run would be a different task.

“the workpaper templates specify the procedures to perform”

Does a single template contain multiple procedures?

From the blog post: Yes. The cash template lists 31 procedures across 7 tabs, and the accounts receivable template 31 procedures across 8 tabs. The task prompt only says to follow the workbook's procedures; the procedures themselves live in the template.

From the paper and blog post: The rubric scores only about ten selected outcomes per workpaper (13 in the cash sample, 16 in the receivables sample), so many procedures are performed but not individually graded.

“The synthetic client binders, moreover, are designed to provide much of the supporting documentation required to complete these procedures, while limiting common errors and ambiguities in the supporting information”

Why limiting but not eliminating? Is it to make it more reflective of real life?

From the paper: Both reasons appear. The appendix presents leftover imperfections, such as wrong account coding and inconsistencies between documents, as realistic. The limitations section says resource constraints and the size of the binders kept them from finding and fixing every issue. Rubrics skip the parts imperfections affect.

Inference: So it is partly deliberate and partly unavoidable. Because affected parts aren't graded, the benchmark mostly doesn't reward handling messy data. There are exceptions: the blog's cash rubric rewards flagging a conflict between two January bank statements, which suggests some inconsistencies were planted on purpose.

“Completed workpapers are graded against rubrics covering an auditor-guided sample of data points, calculations, and conclusions, using reference answers prepared and/or reviewed by CPA-licensed auditors.”

Are the reference answers numbers or free-form text? If it's the latter, how do you grade them?

From the paper: Both. Numeric values are pulled from specific rows and compared with the reference by code. Where a table can be filled correctly in several ways, the table content goes to an LLM judge (GPT-5.6 Luna). Criteria are written to accept substantively equivalent answers.

From the blog post: The cash rubric mixes exact checks, such as operating cash of $3,077,758.05 agreeing to the trial balance, with judgment checks, such as marking a confirmation follow-up as not evidenced instead of claiming it was sent, and keeping the overall conclusion open.

Inference: The paper reports no agreement rate between the LLM judge and CPAs, so grading error on the text checks is unmeasured.

“where reviewers exercise judgment about what to examine rather than systematically reperforming every procedure or checking every cell”

How do they determine what to examine?

Background knowledge: In practice, reviewers focus on high-risk areas, large or unusual amounts, significant estimates and judgments, agreement with the financial statements, and the conclusions. They review less experienced preparers more closely.

From the paper: For FAB, a senior auditor wrote example rubrics for core areas, choosing the data points, calculations and conclusions considered vital; an AI model copied that structure to the other tasks; and CPAs revised the results against model outputs. Evaluation is also limited to sections with well-defined ground truth that binder imperfections don't affect.

Inference: So two filters apply: importance and ease of grading. The second can tilt rubrics toward balances, which are easy to check, and away from judgment.

“The rubrics contain an average of ten scored checks per workpaper, and a workpaper receives a passing score when all applicable criteria are met”

What are some examples of the relevant checks?

From the blog post: The cash rubric has 13 checks in three groups.

• Amounts: operating cash agrees to the trial balance ($3,077,758.05); confirmed operating and payroll balances are recorded; deposits in transit are zero; check 41001 is recorded as clearing on 2026-01-16; all cash is at one bank; uninsured deposits are $2,827,758.05, the balance less the $250,000 FDIC limit.
• Professional judgment: no unsupported claim that a follow-up was sent; the overall conclusion is kept open; the conflicting January statements are described accurately.
• Traceable documentation: every comment on the lead sheet, confirmation and reconciliation tabs sits on an identifiable account row.

The receivables rubric (16 checks) includes footing the aging buckets, recomputing the 1 to 30 day reserve, and totaling only the confirmations actually sent. The journal-entry completeness rubric (7 checks) includes tracing total debits of $10,713,435.26 from the source system through the extract.

“it fails under a strict all-or-nothing grading metric”

Why do they have an all-or-nothing metric as opposed to something more granular?

From the paper: It mirrors workpaper review: a workpaper that misses one vital point can't support an audit conclusion. They also report partial credit.

Inference: Partial credit hides differences, since every model scores between 86% and 96%. All-or-nothing turns small per-check differences into large ones.

My arithmetic: With 10 independent checks, 96% accuracy per check gives 0.96 to the 10th power, about 66% strict pass, and 88% per check gives about 28%. That roughly matches Opus (95.8% partial, 69.2% strict) and Gemini 3.8 Flash (87.9% partial, 27.5% strict).

Inference: The weaknesses: checks aren't weighted, so a documentation check counts as much as a wrong reserve, and workpapers with more checks are harder to pass. Pass rates by check type, or severity weights, would be more informative.

“An engagement binder contains over a hundred financial documents that describe snapshots of the same shared underlying business activities”

What are some examples of the most common kinds of documents?

From the paper (Figure 4) and the blog post (sample binder):

• Planning: scope, materiality memo, risk assessment, selection plan, procedure register.
• Core client records: chart of accounts, current and prior-year trial balances, the full journal entry population, the general ledger.
• Account records: receivables aging, allowance workbook and January cash receipts; payables aging, vendor invoices and January payments; inventory listings and count results; sales invoices and shipping evidence; payroll registers; loan agreements; fixed asset schedules.
• Bank statements: one per account per month. The cash sample had 26 (13 months for two accounts).
• Financial statements: balance sheet, cash flow statement, notes, and the prior year's audited statements.
• Third-party evidence: bank, loan and customer confirmations, and an inventory count observation.

Inference: By file count, monthly bank statements are probably the largest single category.

“At a high level, the graph contains three types of nodes. Source nodes encode user-defined inputs or values sampled from predefined distributions, conditionally sampled nodes draw values from distributions determined by the values of their parent node(s), and deterministic nodes apply accounting rules or calculations.”

What are examples of quantities that would fall in each category and why?

From the paper: Figure 2: industry is a source node; revenue and the payroll-to-revenue ratio are sampled given industry; payroll is revenue times the ratio.

Inference: More examples, split by what is random in real life versus what accounting rules force:

• Source nodes set the scenario and have no parents: industry, fiscal year-end, number of bank accounts, number of customers.
• Conditionally sampled nodes carry realistic variation that depends on context: revenue given industry; how fast customers pay given industry; historical loss rates by aging bucket; the mix of adjusting-entry types; raw-material waste given manufacturing. The private priors feed these.
• Deterministic nodes enforce accounting identities so documents agree: ending bank balance equals beginning balance plus deposits minus withdrawals; receivables equal the sum of open invoices; the allowance equals each aging bucket times its loss rate; uninsured deposits equal the balance minus $250,000.

What are “auditor-developed materials”?

Inference: Documents the audit team creates, as opposed to the client or outside parties: planning memos (materiality, risk assessment, selection plans), workpaper templates, confirmation control logs, procedure checklists and prior-year workpapers. In the binder these are the planning folder and the blank templates. The blog's sample binder also has an auditor-controlled receivables evidence file in the direct-to-auditor folder, which logs the confirmations the auditor sent and received.

How is “third-party evidence” different from “public information”?

Background knowledge: Third-party evidence is about this client and comes from an outside party that deals with it, ideally sent straight to the auditor: bank confirmations, loan confirmations, customer confirmations. Public information is generally available and not about the client: the FDIC insurance limit, interest rate benchmarks, industry statistics, a bank's credit rating, accounting standards.

From the paper: Binders keep third-party evidence in its own folder (direct_to_auditor), separate from client records.

From the blog post: The cash sample uses both: a bank confirmation (third-party) and a deposit insurance reference applying the $250,000 limit (public). The agent noted it had no public evidence on the bank's financial condition and left that procedure open.

“An LLM then proposes candidate variables from these drafts, and three auditors independently assess each variable from its description and propose additions.”

What do they mean by candidate variables? Examples?

Inference: Candidate variables are the quantities needed to generate a document realistically. For a receivables aging: number of customers, invoice count, the mix of payment terms (net 15, 30, 45, 60), how far past due invoices are, and loss rates by aging bucket. For bank statements: deposits per month, the share of ACH payments, transfer patterns. "Candidate" means not all were kept or given private priors; the final private set was six ratios, sixteen record features, and two summaries.

“In each round, the generator produces new binders”

New binders, as in completely new or updated? Why?

From the paper: Both appear. Each round the generator produces new binders, and after revising the graph they regenerate the affected documents for further review.

Inference: After each round they fix the graph and sample again. Fresh samples matter because a rule that works for one sampled company can fail for another, for example by producing a negative month-end balance only for some values. Regenerating only affected documents spares reviewers from rereading files that didn't change. Because every document comes from the graph, one fixed rule updates every document that uses it.

“A senior auditor reviewed the proposed ground truth answers, adjudicated disagreements, and corrected errors”

Disagreement in the event that multiple auditors had reviewed the proposed ground route? Or some other kind of disagreement?

From the paper: At least one CPA "completed and/or reviewed" each workpaper, so some workpapers had more than one auditor, for example one completing and another reviewing. The senior auditor settled disagreements between them.

Inference: The likely cases are two auditors reaching different answers on a judgment item, or a reviewer disputing a preparer's answer. The paper reports no agreement figures, saying feedback was collaborative and spread across revision cycles. So we don't know how often auditors disagreed, which matters most for judgment-heavy areas like receivables.

“Subsequently, the rubrics were checked and revised by CPAs against multiple model submissions to assess their fairness and robustness.”

What do they mean by "against multiple model submissions" here?

From the paper: CPAs ran the draft rubrics on workpapers completed by several models and revised them where the grading was unfair or fragile.

Inference: This tests the grader on real outputs. It catches false failures, where a correct answer worded or placed differently fails (for example, the grader reads the wrong row), and false passes, where a wrong answer gets through. The risk is that tuning rubrics on model outputs fits them to how those models answer. Tuning on some models and reporting on others, or holding out tasks, would guard against that.

What is the difference between rubric and the ground truth?

From the paper: The ground truth is the complete reference workpaper prepared or reviewed by CPAs. The rubric is about ten scored checks drawn from it, plus the rules for comparing an agent's answer, such as how to find the row and when to use the LLM judge.

Inference: The ground truth is the answer key for everything; the rubric is a sample of it. That is why the paper warns that a passing workpaper may still contain errors outside the rubric.

“eight independent runs per model-task pair”

Why eight runs? Also, what are the different sources of inconsistent behavior across runs?

My arithmetic: At the reported mean costs, all 3,960 main runs cost about $7,470 in model fees, so cost was probably not the limit.

Inference: Eight gives a pass^k curve out to k = 8 and a per-task success rate in steps of 12.5 percentage points, which is coarse but usable. Sources of variation:

• Sampling: models decode with randomness; Kimi K3 ran at temperature 1.0 and the others at their defaults.
• Server non-determinism: batching and floating-point effects can change outputs even with fixed settings.
• Path dependence: an early choice, such as which file to open first, changes every later step of a long run. This is probably the largest source.
• Infrastructure: timeouts and retries.
• Grading: the LLM judge is itself random, so part of the variation is in the grader.

“We evaluate all eleven models in a simple agent harness where the only tool provided to the agent is a bash tool, which executes shell commands”

How does it process PDFs if it only has access to a bash tool? Grep?

From the paper: Bash runs programs installed in the container. The system prompt lists pdftotext and pdftoppm (command line), pypdf and pdfplumber (Python), tesseract for text recognition on scans, openpyxl and pandas for spreadsheets, python-docx for Word files, LibreOffice, and ripgrep for search.

From the blog post: Opus converted every planning PDF to text with pdftotext into one file, searched it with grep, read and wrote spreadsheets with openpyxl, and recalculated formulas with LibreOffice.

Inference: Bash returns only text, so the model can't look at a rendered page. Scanned documents need text recognition, and complex PDF tables can lose their structure when converted.

Is 69.17% the average across the 8 runs or is that pass@1?

From the paper: Both, because they are the same number here. Table 1's strict pass rate is averaged across tasks and runs, and that equals the k = 1 value in Table 5.

My arithmetic: 249 of 360 runs.

“auditors compared workpapers completed by agents with reference workpapers completed by auditors and reperformed a limited subset of procedures.”

Which limited subset, and based on what?

From the paper: Not specified. The purpose was to understand failures and check whether the rubrics were fair.

Inference: Most likely the procedures where an agent's answer differed from the reference, since those are the ones you would check to decide whether the agent or the rubric was wrong. Because the sample wasn't random and no counts are given, the four failure types describe what goes wrong but not how often.

“Failure to consider the effects of changes included cases where agents added information to the workpaper without assessing its downstream impacts (e.g., adding numeric details to a table when they should not have been included in the corresponding column total)”

I don't really understand this example.

Inference: Workpaper tables often end with a SUM formula, and other cells use that total; for example, the outstanding checks total feeds the bank reconciliation. If the agent adds a row with an amount meant only as information, such as a check written in January or a subtotal, inside the summed range, the total changes. The reconciliation then stops agreeing and every cell that uses the total is wrong, even though each row looks correct by itself.

From the blog post: The cash sample shows the setup: the agent added a row for all January debits tested ($122,426.95). If that row had sat inside a summed range, it would have been counted twice. The blog lists only three failure types and leaves this one out.

“Insufficient retrieval” can’t really be distinguished from inaccurate retrieval, right?

Inference: They can be distinguished, but only from the run log, not from the final workpaper. As the paper defines it, insufficient retrieval is stopping too early: the right document was in the binder but never opened, and the agent concluded the information didn't exist. That is a recall failure. Inaccurate retrieval would be opening the wrong document or misreading the right one. The other failure types differ as well: weak evidence weighting means the agent found the sources but chose the less reliable one, and completing with insufficient information means the document truly wasn't there. A rubric sees only the final cell, so separating these requires reading trajectories, which is what the auditors did.

“Insufficient retrieval”

Could the problem of insufficient retrieval be solved by modulating the number of model calls? Is there a better way to solve it?

From the paper and blog post: The cap is 200 calls, but agents stop well before it: Opus used 38 calls on cash and 64 on receivables, and median runtimes are 7 to 12 minutes. More reasoning does help: GPT-6 Astra rises from 23.6% strict pass at low effort to 71.1% at maximum.

Inference: The agent stops by choice, so raising the cap won't help. Better fixes target the decision to stop:

• Require an inventory step: list every file and read the binder's package index first (the sample binder has one).
• Require evidence of absence: before writing "not available," the agent must show the searches it ran across all files.
• Convert every document to text up front and provide a search tool, so searching is cheap.
• Add a checker that challenges every "not available" or "unable to conclude" entry.

“Inadequate evaluation of evidence sources included cases where agents vouched amounts to client-provided documents rather than third-party evidence”

Why is this a problem?

Background knowledge: Evidence from the client is only as reliable as the client. A client-prepared cash receipts report shows what the client recorded, not what the bank received, so using it to test receivables is circular. A client inflating receivables could support them with a fabricated report. Bank statements, and especially confirmations sent directly to the auditor, come from a party independent of the client.

From the blog post: The receivables sample shows the cost. Before applying the 5% loss rate to the 1 to 30 day pool, Opus subtracted $136,102.85 of February receipts taken from the client's own listing, which it had not matched to a bank statement. That cut the reserve from $7,593.70 to $3,154.23 and was the one check it failed.

“These estimates exclude senior review and corrective rework and therefore do not establish equivalent time savings in practice.”

But those additional stages are excluded/included in both cases, right (whether it’s an agent or human doing it)? So the time savings should still apply?

Inference: The preparation saving is real: 2 to 12 hours versus 7 to 12 minutes. The paper's caution is that review and rework are not the same size in the two cases.

• Rework: agent workpapers fail strict grading 31% to 73% of the time depending on the model, and receivables fail 96% of the time on average.
• Review: reviewers know a staff auditor's track record but can't yet calibrate trust in an agent, and the paper says agent documentation is harder to review.

Net saving equals preparation time saved minus extra review and rework, and only the first part was measured. On journal-entry completeness (94.5% pass) the saving is close to the full preparation time; on receivables (4.2%) nearly every workpaper needs correction, so much of it could be lost.

“FinancialAuditBench includes 45 review tasks evaluated across ten models, with all models receiving the same starting workpaper”

What do they mean by “starting workpaper”?

From the paper: For each review task, one completed workpaper from a randomly chosen model in the main experiments. Every reviewing model gets that same workpaper and the same supporting files, so differences in results come from the reviewer alone.

“Claude Opus 5.5 achieves the largest improvement, increasing strict pass from 28.89%”

Where did the 28.89% number come from? Is this representative?

From the paper: It is the strict pass rate of the 45 starting workpapers before review: 13 of 45 pass, with 87.34% partial credit.

My arithmetic: Averaged over the 11 models, main-experiment strict pass is about 42.6% and partial credit about 90.0%. Random draws would be expected to give about 19 passing workpapers, not 13. That is a little under two standard deviations below expectation.

Inference: So the starting set is weaker than typical, whether by chance or because of how the draw was done (the paper doesn't say whether it drew a random run or a random model). A weaker start leaves more room to improve, so the review gains probably overstate what review adds to typical work. Also, Opus after review reaches 62.22%, below its own 69.17% when completing from scratch.

Why is the discussion on review tasks so much shorter than the discussion on completion tasks?

From the paper: The review experiment is smaller: one run per model per task (450 runs against 3,960), ten models (Fable 5.1 not included), and one starting workpaper per task. It has no reliability or failure analysis.

Inference: Review is a secondary result, and with one run per model the authors had less they could say with confidence. Two things are missing: how often reviewers broke correct work (the paper says some scores dropped but gives no count), and which errors reviewers caught or missed.

“To address these limitations, we restrict evaluation to workpaper sections that have well-defined ground truth, focusing on procedures that are common across financial statement audits in multiple industries.”

What are examples of those kinds of procedures?

Inference, guided by the blog's sample rubrics: Procedures whose answer follows from the evidence and arithmetic:

• Agreeing an account balance to the trial balance and the financial statements.
• Agreeing bank balances to the bank confirmation and recomputing the reconciliation.
• Footing (adding up) an aging report and recomputing a reserve from given loss rates.
• Computing uninsured deposits.
• Proving journal-entry completeness by rolling account balances forward.
• Recomputing interest and depreciation.
• Testing cutoff against shipping dates.

Left out would be open judgments, such as whether management's loss-rate assumptions are reasonable, whether there is a going-concern problem, or what materiality should be.

“The overall process of a financial statement audit spans the general phases of planning, risk assessment, substantive fieldwork, concluding, and reporting”

What is the difference between these phases?

Background knowledge:

• Planning: understand the client, set materiality, staff the engagement and set the timeline.
• Risk assessment: identify where material misstatements could occur, by account and assertion, including fraud risks, and understand the client's controls.
• Substantive fieldwork: perform the planned tests, gather evidence and document it in workpapers. This is FAB's scope.
• Concluding: compare uncorrected misstatements with materiality, review events after year-end and going concern, obtain management's written representations, and complete senior review.
• Reporting: issue the opinion (unmodified, qualified, adverse, or a disclaimer) and report findings to the board.

From the paper: Fieldwork takes most of the effort, which is why FAB targets it. The planning folder stands in for the first two phases.

“Each firm organizes its workpapers differently and uses its own respective testing procedures, review processes, documentation processes, etc. that comply with applicable generally accepted auditing standards”

Is there any official criteria listed in any regulations or law somewhere?

Background knowledge: Yes. The standards say what must be documented, not how the spreadsheet is laid out.

• US private-company audits follow the AICPA's generally accepted auditing standards (GAAS), codified as AU-C sections. AU-C 230, Audit Documentation, requires enough detail for an experienced auditor with no prior connection to the audit to understand the procedures, evidence, conclusions, and who did and reviewed the work.
• US public-company audits follow PCAOB standards under the Sarbanes-Oxley Act. AS 1215 sets documentation rules, including assembling the final file within 45 days of the report and keeping it for seven years.
• Internationally, ISA 230 is the equivalent.
• Firm-level quality management standards govern review processes.

Enforcement comes from PCAOB inspections, AICPA peer review, and state boards of accountancy, which license CPAs.

“In this review setting, for each task, a completed workpaper from a randomly chosen model in the main experiments serves as the starting workpaper for every reviewing model.”

Why from a model as opposed to real life for review?

Question: Why from a model as opposed to real life for review?

Inference: Four likely reasons:

• Privacy: real completed workpapers are confidential, which is the problem the paper exists to avoid.
• Cost: human-prepared workpapers on the synthetic binders need auditor hours, and the paper says resource limits already restricted how many auditors completed.
• Relevance: the intended use is AI preparing and someone reviewing, so model-made errors are the realistic input.
• Availability: the model outputs already existed.

Costs of the choice: model errors may differ from human ones, and because the starting workpaper came from an evaluated model, a reviewer sometimes reviews its own model's output, which may make errors harder to see. An alternative is planting known errors in correct workpapers, as in the Forus perturbation post in your Applications tab, which shows exactly which errors were caught.

Any reporting of two-phase pipeline with the best model doing both completion and review?

From the paper: Not reported.

Inference: The closest numbers are Opus completing from scratch (69.17%) and Opus reviewing a weaker random workpaper (28.89% raised to 62.22%). An Opus-then-Opus pipeline starts at 69% and could go higher, but self-review may share the preparer's blind spots, and review can damage correct work. The experiment to run: complete with model A, review with A and with a model from another family, and report per-task changes in both directions.

Why is Astra reported separately?

From the paper: GPT-6 Astra appears in two side experiments, the reasoning-effort ablation and the computer-use comparison. No reason is given for choosing it.

Inference: For computer use, the ChatGPT desktop app ran GPT-6 Astra, so the bash comparison had to use the same model. For the ablation, OpenAI exposes five effort levels, which makes it easy to sweep.

From the paper (Table 7): The ablation matters more than its placement suggests. At maximum effort Astra reaches 71.11% strict pass, above Opus 5.5's 69.17% at medium, at $8.27 and 25 minutes per run. The main leaderboard compares models at one effort setting (medium for most), so it ranks configurations, not each model's best performance.

“The default caching policy is used for all models.”

What is the default caching policy?

From the paper: Not defined.

Background knowledge (may be out of date): It refers to prompt caching: the provider stores its processing of a prompt's opening portion and charges less, and responds faster, when the next request repeats it. Agent loops benefit a lot, because every call resends the whole growing conversation. Defaults differ: OpenAI caches long prompt prefixes automatically, Anthropic's API has required the caller to mark cache points, and providers reached through OpenRouter vary.

Inference: Caching affects cost and speed, not accuracy. Because defaults differ by provider, part of the cost column in Table 1 reflects caching policy rather than model efficiency.

“Other parameters not explicitly outlined here retain their default values.”

What would those other parameters be?

Background knowledge: Maximum output tokens per call, top-p and top-k sampling limits, frequency and presence penalties, tool-choice settings, whether parallel tool calls are allowed, reasoning-token budgets and stop sequences.

From the paper: Only reasoning effort and, for Kimi K3, temperature (1.0) were set. Requests had a 600-second timeout and up to two retries.

Inference: Defaults are each provider's choices, so they aren't comparable across providers. The one most likely to matter is the output-token limit, since a low default can cut off a long command or reasoning trace.

“The agent harness does not perform context compaction or summarization.”

What happens if the context window limit is reached?

From the paper: Not stated. Runs end at the completion signal or at 200 calls, and fewer than 1% failed from external errors such as API errors and were rerun.

Inference: Without compaction, the conversation grows with every tool output. If it passes the model's context limit, the API rejects the request, and the run either ends without a completion signal (likely scored as a failure) or is treated as an external error and rerun. The paper doesn't say which. Agents mostly avoid the limit themselves: the blog's traces show them printing only the start of outputs and filtering with grep.

From web search: GPT-5.6 models have context windows of about 1 million tokens.

Inference: With limits that large, overflow is probably rare, but models with smaller windows are penalized more by the no-compaction choice, and quality can drop as context fills even below the limit.

“n this work, rubrics are written by auditors who check the presence, absence, and/or correctness of specific cells or rows deemed vital in each workpaper.”

My understanding was that all cells will be present but the values in them may differ. Am I missing something?

From the paper: Columns are fixed but rows are not: agents add and remove rows. So whether a row exists is part of the answer.

From the blog post: The cash and receivables rubrics have both kinds. Presence: the payroll account must appear as a confirmed $0.00 balance, and check 41001 must appear as clearing on 2026-01-16. Absence: the confirmation total must include only the three requests actually sent, and a follow-up must not be recorded as done without evidence that it was sent.

Inference: Absence checks catch content the agent should not have written, usually claims the evidence doesn't support.

“Agents may add or remove rows from each table during each task.”

Why would the agent do this?

Inference: A template can't know in advance how many items a client has. The number of bank accounts, reconciling items, sample selections, exceptions and open issues depends on the evidence, so the agent adds a row for each item and removes unused placeholder rows.

From the blog post: In the cash sample, the agent added a row for the lender's loan confirmation, two rows for January exceptions on the cutoff tab, and total rows, and noted that it left all columns and headers unchanged.

“Our provided rubric criteria are then applied to each table via a keyed approach that first locates the column of the table to be evaluated and then extracts values from target row(s). This allows grading to accommodate changes in row positions without relying on fixed cell coordinates.”

I didn’t understand this.

Inference (explaining the paper's method): A fixed-coordinate grader says "check cell F14." If the agent inserted a row above, the value is now in F15 and the grader fails a correct answer. A keyed grader instead says: on the bank confirmation tab, find the column headed "Per confirmation," find the row whose account is ****0780, and read that value. It works like looking up a value by column name and row label (INDEX/MATCH in Excel, or a WHERE clause in SQL) instead of by address. It works because the template fixes the columns and each row has an identifying key, such as an account number. When the right row can't be identified that way, because several valid layouts exist, the table goes to the LLM judge.

“GPT-5.6 Luna with medium reasoning effort is used as the model for LLM-based evaluation.”

Why Luna?

From web search: GPT-5.6 Luna is the lowest-cost model in OpenAI's GPT-5.6 series, roughly the "nano" tier of earlier families, with a context window of about 1 million tokens.

Inference: It is a sensible choice. Grading runs on many checks across thousands of runs, so cost matters; the judging task is narrow (compare extracted table content with a reference); and Luna is not one of the eleven evaluated models, so no model grades its own output. Its siblings Sol and Terra are evaluated, though, so a bias toward OpenAI-style answers can't be ruled out, and the paper reports no agreement rate between the judge and CPAs.

"Access only the designated binder folder and its subfolders. Do not read, search, or modify files outside that folder, access other binders, or use websites or external data sources.”

Why is this restriction in the prompt as opposed to being an actual hard constraint somehow?

From the paper: This instruction comes from the computer-use experiment, where the ChatGPT desktop app operated the computer with mouse and keyboard.

Inference: In the bash setting the restriction is enforced by the environment: an isolated Docker container with no network. A desktop app controlling a real computer has whatever access the signed-in user has, and a screen-driving agent can open any window or browser. Enforcing the rule would need a dedicated virtual machine with only that binder and no network. They may have done that; the paper doesn't say, and the instruction implies other binders were on the same machine. A prompt is not a guarantee, and compliance can only be checked afterward. The risk is contamination, for example reading another binder's materials.

What is the purpose of the computer use experiment in the first place?

From the paper: Audit work often happens in enterprise applications that agents can't control through code, so they have to use the screen as a person would. The experiment asks whether working through the screen hurts quality. On 8 tasks from one binder, both using GPT-6 Astra, computer use reached 50.00% strict pass and 92.84% partial credit, against 54.69% and 93.58% for the bash agent. It took 283 minutes against 65 and left many windows open.

My arithmetic: 50.00% is 4 of 8 tasks, which implies a single run, while 54.69% is 35 of 64, which implies eight runs. So one run is being compared with an eight-run average, too few to tell the two apart on quality.

Inference: The clear finding is speed: computer use was about 4.4 times slower.

How would you improve this system if you were building it today?

Inference: One or two changes per component:

• Tasks: add multi-turn tasks where the agent can request documents from a simulated client, since requesting and following up is a core audit skill the paper says is missing. Add tasks where key evidence is deliberately missing or contradictory, and score whether the agent leaves the item open; this tests the fourth failure type directly.
• Grading: weight checks by severity; report pass rates by check type (amounts, judgment, documentation); score how easy the workpaper is to review, as the paper proposes; and publish agreement rates between the LLM judge and CPAs and between CPAs.
• Experimental design: report each model across effort levels as a cost-versus-accuracy curve instead of one setting (Q62); add confidence intervals, since each area has only 3 to 6 tasks; and test preparer-plus-reviewer pipelines (Q61).
• Validity: test realism blind, by asking auditors to tell synthetic binders from real ones, and keep a private held-out set of binders so the public benchmark doesn't end up in training data.

“A support associate would glance at the account, check the spelling, and move on.”

How would the support associate know the account information? Are they cross-referencing based on the phone number?

From the post: The post doesn't say how the associate finds the account.

Inference: The account is usually pulled up before verification starts, most often by caller ID, which matches the calling number to the number on file, or by an identifier the caller gave earlier, such as an account number. The phone number is only a lookup key, not proof of identity, because caller ID can be spoofed. That's why the associate still checks the name.

“To evaluate performance under these conditions, we built an internal benchmark using domain-specific customer service audio.”

What are the main performance metrics for this kind of data? How are they defined and measured?

From the post: Only one metric is named: utterance error rate (UER), how often an utterance has a meaning-changing error.

From web search (μ-Bench blog and README): Sierra's open benchmark also reports word error rate (WER), which is substituted, deleted and inserted words divided by the words in a human reference transcript, and 95th-percentile latency to a complete transcript. The blog's case for UER is that WER counts a dropped "uh" the same as a misheard phone-number digit. An LLM judge decides which errors change the meaning.

Inference: UER is the right primary metric for agents. Its weak point is the judge: I couldn't find how often it agrees with human reviewers.

“And unlike other conversations — where agents can infer intent from a “good enough” transcription — verification is binary: you need an exact match.“

What verification are they talking about here, and why is it binary in this case but not in others?

From the post: Identity verification, such as confirming a caller's name with a bank or insurer.

Inference: It's binary because the transcript is compared with a stored value, and the comparison either matches or doesn't: "Kaitlyn" fails against "Caitlyn." In other turns, the transcript goes to an LLM that can understand the caller's intent despite small errors. Systems can loosen the match, for example by ignoring case or matching on sound, but each loosening makes it easier for the wrong person to pass.

“If context shifts mid-conversation, the challenge gets harder.”

Shouldn't we default to using a multilingual transcription provider then? Any major downsides to that?

From web search (μ-Bench blog): No provider was best in every language, and the most accurate one was among the slowest.

Inference: A multilingual model is a good fallback but a weak default, for two reasons. It usually trails a model tuned for one language on that language. And it has to guess which language it's hearing, which is hard on short inputs like a spelled name.

“Standard transcription pipelines process audio in isolation, without awareness of what the conversation is about or what the caller is likely to say next.”

Similar to the way we give LLMs context at the start of their prompt to inform subsequent generation, are there any architectures/models for voice transcription that can do the same thing?

From web search (Deepgram): Deepgram's Nova-3 accepts up to 100 key terms per request to improve their recognition, without retraining.

Background knowledge: Yes, three kinds. Conventional speech recognition can favor a supplied word list, like Deepgram's. Models with a text decoder, such as Whisper, accept a text prompt treated as the preceding transcript, which is closest to your LLM analogy. Speech LLMs, where an audio encoder feeds a language model, take context exactly like an LLM prompt. The shared risk is that the model outputs a favored word the caller didn't say.

“Instead of picking one provider and managing its weaknesses, we built an ensembler that queries multiple providers in parallel and combines their outputs.“

A major consideration in voice AI is latency. If we're introducing an ensemble layer, what happens to our latency?

Inference: Because providers run in parallel, the added latency is roughly the slowest provider you wait for plus the ensembler's own processing, not the sum of all providers. Two design choices keep it small. A deadline lets the ensembler use whatever has arrived and drop slow providers. A fast path accepts the transcript immediately when providers agree, and runs the slower resolution step only where they differ.

“Cross-reference outputs and identify where providers agree or diverge“

What does this mean concretely?

Background knowledge: The classic method, ROVER (1997), aligns transcripts word by word and votes at each position.

Inference: Sierra likely aligns the transcripts to find the specific words where providers differ, such as a name returned as "Caitlyn," "Kaitlyn" and "Katelyn." It then resolves only those words, using information a vote ignores: each provider's accuracy in that language, confidence scores, and the name on file. Giving an LLM the candidates and the context is a plausible way to do that last step.

“Incorporate signals from earlier turns in the conversation.”

What does this mean concretely?

From the post: The only earlier-turn signal the post mentions is the conversation's language. Its name and address examples come from the customer record, not earlier turns.

Inference: The two most useful signals are probably the agent's last question, which predicts the type of answer (a date, a spelled name, digits), and entities already mentioned, such as an order number or product name that's likely to come up again.

“On our internal benchmarks, we have found that ensembling can cut utterance error rate (how often an utterance has a meaning-changing error) by ~25% on average versus the best single provider, and by up to 37% in languages with more headroom for improving transcription.“

How to weigh this performance improvement versus the presumed latency increase?

Inference (the numbers are my illustration): Compare expected time, because errors cost time too. If ensembling cuts errors from 8% to 6% of utterances, and each error costs a 6-second re-ask, it saves about 120 milliseconds per utterance on average. So it pays off if it adds less than that. Failed verifications and transfers to a human cost far more than a re-ask, so the case is strongest on high-stakes turns like verification and weakest on small talk.

“Rather than asking the transcription model to guess from the full space of possible utterances, we feed it context from the conversation.”

Is the context fed to the transcription model or the agent in a post-processing layer?

From the post: To the transcription side, not the agent. The ensembling diagram also shows context going into the ensembler.

Inference: The context is probably used twice: as bias terms for each provider, and by the ensembler when choosing among their outputs. One risk is that biasing toward the value on file makes a near-miss more likely to be transcribed as an exact match, which weakens identity checks.

“For our financial services agents, context-aware transcription improved input verification rates by over 25%”

What is input verification?

Inference: It most likely means checking that a value the caller says, such as a name, date of birth or account digits, matches the record, with the rate being the share of attempts that pass. Two things are unclear: whether "25%" is a relative increase or percentage points, and whether some of the extra passes are callers who shouldn't have passed. The false-accept rate is the missing number.

“Sierra voice agents have improved resolution rates by up to 1%, which translates to tens of thousands of resolutions a week, and reduced major transcription errors by up to 15%.”

Why is the improvement in transcription errors so much higher than the improvement in resolution rates?

Inference: The 15% is a relative cut in errors, while resolution is measured over all calls, and most calls have no major error. Of the errors that do occur, many don't change the outcome because the agent asks again or understands anyway. And most unresolved calls fail for other reasons, such as policies that require a human. As a rough check, if transcription errors caused failures on F% of calls, a 15% cut would recover about 0.15 × F points, so a full point would need F near 7%. That suggests 1% is a best case.

“the system dynamically reconfigures the transcription pipeline — selecting a different ensemble of providers optimized for that language”

How is the ensemble selected?

From web search (μ-Bench blog): Sierra benchmarks 79 locale variants across 42 languages and more than 13 providers internally, and says this is what lets it pick models per language.

Inference: Most likely, the best ensemble for each locale is chosen offline on accuracy, latency and cost, and stored in a lookup table. When the system detects the caller's language, it switches to that entry.

“No amount of ensembling or context injection will produce a reliable transcription from unintelligible audio. In those cases, the best response is to ask for clarification, just as a human would.”

Is this done using log probs or something else?

Inference: Probably several signals rather than log probs alone. Speech recognition models do return per-word confidence scores, their equivalent of log probs, but those are often poorly calibrated. They're likely combined with disagreement between providers, which the ensemble provides for free, and with checks such as a date that's impossible or a value that doesn't match the record.

How would you improve this system if you were building it today?

Inference: I'd make one change at each stage of the pipeline:

• Routing. Ensemble only on high-stakes turns such as verification and data capture, and use the fastest accurate single provider elsewhere (#9).
• Decoding. For structured inputs, restrict output to the expected format, such as digits, dates or letters, and accept the phonetic alphabet ("B as in boy").
• Verification. Keep "what was said" separate from "does it match," log how much each match relied on biasing, and track the false-accept rate (#10, #11).
• Learning. Store provider disagreements with human-reviewed corrections, use them to train the ensembler, and alert when the disagreement rate jumps, since that often means a provider has degraded.
• Evaluation. Report end-to-end task success, for example from τ-voice simulated calls, alongside error rates, because #12 shows transcription gains don't translate directly into resolutions.

“Picking a pharmacy. How hard could that be?”

Why is Forus involved in picking the pharmacy at all? Isn’t that something Maria would do herself?

From the post: Forus handles the step after the prior authorization (PA) is approved. Maria's preference is one input, and the agent can “request patient preferences.”

Inference: For a specialty drug the choice isn't free. Her plan often requires a specific specialty pharmacy, and the manufacturer may limit which pharmacies can stock the drug. Most patients don't know these limits. Also, the prescriber's side sends the prescription, and Forus does that work for practices. Maria's preference only decides among the valid pharmacies.

“[W]hich eligible pharmacies are in-network can change without warning. Often, we find out when a routing fails.”

No API connection for checking in-network?

From web search: Drug Channels says there is “no master database” of BIN/PCN/Group codes.

Inference: Contracts between specialty pharmacies and PBMs (the companies that process drug claims for insurers) are private, change on their own schedule, and have no public data feed. The most reliable signal is a rejected claim, which is why a failure is how Forus finds out.

“Insurance routing depends on BIN, PCN, and Group. We may only have two of those three codes.”

What’s the difference between these three and why might you only have two of them?

From web search (Drug Channels): The BIN identifies the insurer or PBM processing the claim. The PCN identifies the benefit package within that payer. The Group identifies the specific plan, such as one employer's plan.

Inference: There are two ways to end up with only two codes:

1. The code doesn't exist. Some processors don't use a PCN, and some cards have no group number.
2. The code exists but wasn't captured, for example because the EHR has the wrong card, the photo is blurry, or a form field was skipped.

“Two pharmacies may both be valid choices, so accuracy against a single label can mislead.”

Spectrum of validity or binary?

From the post: Validity is yes-or-no for each pharmacy. The issue is that several pharmacies can each be valid, so the right answer is a set.

Inference: It's binary for correctness: a pharmacy is either in network and able to dispense the drug, or it isn't. It's a spectrum for quality: speed, distance, copay and the patient's preference. Re-route rate only measures the binary part, so a valid but slow pharmacy still counts as a success.

“Unresolved cases went to manual review.”

What does manual review entail?

From the post: Forus's clinical experts do it, using the same tools the agent later got: searching pharmacies, reading documents, checking insurance, contacting the patient and scheduling calls.

Inference: A reviewer likely reads the case documents and works out which pharmacies the plan and manufacturer allow. When unsure, they call the PBM or the pharmacy to confirm network status. They check the patient's preference if it matters, then send the prescription. The phone calls are what make it slow.

“14-case if/else cascade for retail drugs and ~30 hardcoded insurance rules for specialty drugs”

What's the difference between retail drugs and specialty drugs? Why would the rules for them differ?

Background knowledge: Retail drugs are cheap, common and stocked almost everywhere. Specialty drugs are expensive, often biologics that need special handling, and payers or manufacturers often restrict them to specific pharmacies.

From the post: The retail rules compare the pharmacy listed in the EHR with past prescriptions. The specialty rules are insurance rules.

Inference: The two rule sets answer different questions. For retail drugs, almost any in-network pharmacy works, so the question is where the patient likes to go. For specialty drugs, the question is which pharmacies the plan allows, and that depends on the insurance codes.

“Our data was tabular with lots of missing values, which is exactly where gradient-boosted trees shine.”

How do they handle text data? Embedding conversions?

From the post: V2 used no free text; all nine features are short structured fields. Free text (letters, notes, patient texts) only came in with V4. LLM extractors pull a pharmacy name out of the text and fuzzy-match it to Forus's pharmacy database.

Inference: The text fields were most likely treated as categories, which XGBoost handles directly, rather than as embeddings. Extraction fits free text better than embeddings because the useful content is one specific pharmacy, and an embedding would blur it.

“We set per-pharmacy confidence thresholds to target >97% accuracy on automated selections.”

1. How is the threshold determined and why is it different for each pharmacy?
2. Reasoning for 97% target?

Inference (1): For each pharmacy, they likely picked the lowest score at which accuracy on held-out data stays above 97%. Thresholds differ because the scores aren't equally reliable. A score of 0.8 for a frequently used pharmacy may be right far more often than 0.8 for a rarely used one.

Inference (2): It's a business tradeoff between the cost of a wrong routing (a patient waiting days) and the cost of manual review, probably benchmarked against how accurate humans are. At 97%, about 3 in 100 automated routings are wrong.

“Instead of a frozen feature snapshot, the model uses a live history of routing outcomes, weighted by recency.”

So the routing outcomes are features?

From the post: Not directly. Matching past routings are summarized into features: the top pharmacy, how much of the evidence agrees on it, how fresh it is, how many routings there are, and how much the rules agree.

Inference: The key change is where the knowledge lives. In V2, the link from insurance codes to pharmacy was built into the model, so it only changed when the model was retrained. In V3, that link lives in the routing history, which updates with every routing. The model only learns how much to trust a given body of evidence, and that changes slowly.

“We search every combination of at least two available features for matching successful routings.”

What does this mean exactly? How are the matching successful routings used?

From the post: They take the fields that are filled in and form every combination of two or more of them. Each combination is a “rule,” for example BIN + PCN. A rule's evidence is the past successful routings that match on those fields. Those routings are weighted by age, summarized into features, and scored by XGBoost.

My arithmetic: Four filled-in fields give 11 rules, which matches the diagram's “10 of 11 matching rules.”

Inference: It's a back-off scheme: when a specific combination has little history, broader combinations still supply evidence.

“The powerset approach substantially outperformed approaches that treated missing features as unknowns.”

By what metric?

From the post: The post doesn't say and gives no numbers.

Inference: It's probably re-route rate, or the share of cases that can be automated at the 97% target. Here is why the powerset approach would win. If “missing” is treated as its own value, a prescription with no Group only matches past cases that also had no Group, which splits the evidence for no real reason. The powerset approach ignores missing fields and uses every routing that shares the fields that are present.

“We keep every matching rule.”

What do they mean by matching rules, and what does it mean to keep them?

From the post: A matching rule is a combination of fields that has past successful routings behind it. Keeping every rule means they don't keep only the most specific one; every rule gets scored, because “when broad and specific rules disagree, that disagreement is useful.”

Inference: Throwing away the broad rules would lose two things. First, a specific rule with 3 routings may be less reliable than a broad rule with 900. Second, disagreement between broad and specific rules signals that something is changing.

“The model uses evidence across the matching rules, including how much they agree.”

How do you measure how much they agree?

From the post: The diagram shows agreement as a count, “10 of 11 matching rules favor RxGate,” meaning the share of rules that pick the same top pharmacy.

Inference: Two variants would probably work better. One weights each rule's vote by how many routings it has. The other compares the most specific rule directly with the broadest one, since the post singles out that split as a sign of change.

“we give recent evidence more weight using a time decay function.”

How are the parameters for the time decay function determined?

From the post: The post doesn't say. Its chart shows a weight near 1.0 for the first week, 0.5 around day 25, and near zero by day 90. It frames the choice as a tradeoff between throwing away useful history and keeping stale routings.

Inference: The parameters were most likely tuned by backtesting: replay past prescriptions in time order and pick the curve that gives the lowest re-route rate. The curve could also be tuned separately for payers whose networks change often.

“For each pharmacy, sum the weights of its historical routing cases. The pharmacy with the highest total becomes that rule’s top pharmacy.“

Why not retain the scores for each pharmacy and use those as features?

From the post: They partly do, in summary form: percent match is the winner's share of the weighted total.

Inference: A separate feature for every pharmacy would tie the model to specific pharmacies. A network change would then require retraining again, which V3 was built to avoid. Your concern still holds: keeping only the winner throws away the runner-up, which matters when the winner has just left the network. One fix is to score each candidate pharmacy within a rule. A cheaper one is to add the gap between first and second place as a feature.

“Time-weighted percent match: How much of the weighted evidence agrees on the winner? A rule at 95% is a much stronger signal than one at 55%.”

What does it mean for a rule to be at 95% versus 55%? What are those percentages measuring?

From the post: The post only defines it: “How much of the weighted evidence agrees on the winner.”

Inference: For one rule, divide the weighted evidence for the top pharmacy by the rule's total weighted evidence. At 95%, nearly all recent routings for that rule went to one pharmacy. At 55%, the top pharmacy barely leads, which suggests two pharmacies are both valid or a switch is underway. Percent match ignores volume, so 95% from 2 routings is weak evidence. That's why routing count is a separate feature.

“One XGBoost model scores each matching rule using its weighted results, evidence freshness, routing count, and the number of fields in the rule, along with agreement across rules. The score estimates how likely that rule’s top pharmacy is to be a valid choice.”

What does it mean for an XGBoost model to score a rule? I don’t really get the statement about how it “estimates how likely that rule’s top pharmacy is to be a valid choice”.

From the post: The highest-scoring rule's pharmacy becomes the prediction, and its score becomes the confidence.

Inference: Think of each rule as a witness that names one pharmacy. XGBoost rates how reliable each witness is, based on its features: percent match, freshness, volume, specificity and agreement. Its output is the probability that sending the prescription to that witness's pharmacy will work. It was probably trained on past prescriptions, with each rule labeled 1 if its pharmacy turned out to be valid and 0 if not.

I’m having trouble reconciling the diagram with the description. Clarify how the XGBoost pipeline works and how the rules fit into it.

From the post: The diagram differs from the text in two ways. It lists routing count as one of each rule's four features and shows agreement separately, because agreement is computed across rules. It also draws a single XGBoost box, but the model scores each rule separately.

Inference: The pipeline runs in this order:

1. Form every combination of two or more available fields; each combination is a rule.
2. Pull each rule's past routings and weight them by age.
3. Compute each rule's features, plus agreement across rules.
4. Score each rule with XGBoost.
5. Use the top-scoring rule's pharmacy as the prediction.

“Each extracted name is fuzzy matched against our pharmacy database to select a specific pharmacy.”

And then what? Is that the final decision or another feature somehow?

From the post: By V4, the extracted pharmacies fed into an 11-step cascade with a fixed priority order: “Doctor note beats insurance rule. Insurance rule beats routing history.”

Inference: The extracted pharmacy was most likely a decision at its own step, not a model feature. If a PA letter names a pharmacy, that step fires and the cascade stops there. This is the weakness the post describes, because a year-old note can override fresher evidence.

“At this point, our cascade had 11 steps. Signals often conflicted. The waterfall resolved them using the priority order we set. Doctor note beats insurance rule. Insurance rule beats routing history.”

I thought they were using the XGBoost model at this point, not the cascade?

From the post: They used both. XGBoost never replaced the cascade: “To improve the cascade, we added an XGBoost model.”

Inference: Through V4, the model was one step in the cascade, and “routing history” in the priority order most likely means the model's prediction. Each piece worked on its own, but the code that combined them was still a fixed priority list. The agent replaced that combining step.

“Those tools let it search pharmacies, read documents (including insurance cards via vision), check dispensing restrictions, look up insurance details, request patient preferences, schedule phone calls, store a pharmacy selection, or escalate to an expert. It reads everything, decides which pieces of information to rely on, investigates further if it needs to, and then acts.“

Do we have any latency constraints? This feels like a time-consuming process.

From the post: The post mentions no latency limit. Its time scale is days: a wrong pick delays treatment by days.

Inference: The time budget is hours, not milliseconds, because this runs in the background after the PA is approved. A few minutes of agent work is tiny compared with a re-route. The real constraints are cost per case and human wait time, since asking the patient or calling a pharmacy adds real delay. A sensible design is to act right away when the signals agree and investigate only conflicts.

“Ask Maria for updated information. Escalate if selection remains unclear.“

What does escalation mean here?

From the post: Escalation hands the case to a Forus clinical expert, the same people who did manual review earlier.

Inference: “Unclear” probably means Maria doesn't reply, or her answer still doesn't lead to a valid pharmacy. The expert then calls pharmacies or the PBM and decides. Ideally the expert gets the evidence the agent already gathered, so they don't start from scratch.

“The first version of the agent performed worse than the waterfall in many cases. But when it got things right, its reasoning reflected the kind of cross-signal thinking that our clinical experts exhibit.”

How much does the latter matter given the former? Is there any good way to combine the strengths of the agent with the waterfall?

From the post: They didn't ship based on reasoning quality. The agent ran in shadow mode for weeks, and its actions were unlocked gradually.

Inference: Overall accuracy matters more, because patients bear the cost of errors. The good reasoning mattered as a sign of potential. The waterfall's flaw is built into its design, since a fixed priority order can't spot stale signals. The agent's errors, by contrast, can be fixed with better prompts and tools. Three ways to combine them:

1. Let the waterfall handle cases where the signals agree, and send conflicts to the agent.
2. Give the agent the V3 model's prediction as a tool.
3. When the two disagree, investigate or escalate.

“We couldn't A/B test with live prescriptions flowing through our system.”

Why not?

From the post: The post gives no reason.

Inference: An A/B test would deliberately route real prescriptions through an unproven system, and each wrong routing leaves a patient without medication for days. The agent's actions, such as texting patients and scheduling calls, also can't be undone. Shadow mode avoids both problems and compares the two systems on the same prescriptions. Its limit is that when the agent picks a different pharmacy, you can't see whether that pick would have worked.

“A denial from a previously valid pharmacy or a provider note pointing somewhere new can prompt it to request a patient preference, suggest a phone call to verify network status, or escalate to a clinical expert.”

How do you actually encode these conditions in code? Would a system that prompts further digging whenever a high-confidence prediction is wrong help with that?

From the post: The post shows no code.

Inference: It probably works in two layers. Ordinary code detects events, such as a pharmacy rejection or a new note, and reopens the case. The LLM then chooses a response from its tools, within guardrails such as a list of allowed actions.

Your idea would help, and it builds on the post's own trigger. One failure on a confident route should flag that route for every pending prescription and trigger a single verification call, instead of letting each patient fail in turn. That fills the gap the post admits: time decay “doesn't tell us that a network has changed.”

How would you improve this system if you were building it today?

Inference:

1. Evidence: treat one failure on a confident route as a warning for every prescription on that route (Q25), and check high-volume routes on a schedule.
2. Decision: let the V3 model handle easy cases and send only conflicts to the agent (Q23).
3. Evaluation: re-route rate misses prescriptions that stall at a valid pharmacy, so also track the time from approval to first fill.

“LLM-as-judge systems have a well-known failure mode: they tend to be overzealous, unnecessarily flagging correct answers.”

Any studies where I can find more details about this besides the source they linked?

From web search: Three more targeted sources:

• VeriFact (Chung et al., NEJM AI 2025) is the post’s own reference [6] and the closest match: an LLM judge checking clinical text against EHRs with the same three labels. It reports the judge “is biased in assigning Not Supported rather than Not Addressed verdicts when compared against humans.”
• Jing et al., “On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation” (NAACL Findings 2025), found Llama 3.1 8B “overly strict” but Claude Haiku “extremely lenient.”
• MiniCheck’s LLM-AggreFact benchmark (Tang et al., 2024) compares fact-checkers that verify claims against grounding documents.

Inference: Which way the bias goes depends on the model and the prompt, so “well-known failure mode” overstates it. Over-flagging is common in strict grounding setups like this one, but it isn’t universal.

“every extraneous flag leads to additional steps”

What additional steps typically?

Inference: The likely sequence is:

• A human reviewer (Forus operations or clinical staff) re-reads the chart to check the flagged claim.
• If the chart is ambiguous or incomplete, someone contacts the provider’s office for clarification or documentation. This is usually asynchronous and can take days.
• The answer is corrected or confirmed, possibly with provider sign-off.
• The form is re-verified and submitted to the payer.

For a false flag, all of this is wasted, because the original answer was right. The main cost is time spent waiting in the queue rather than reviewer minutes, since the PA can’t be submitted until the flag clears. AMA surveys report that PA delays often lead patients to abandon treatment (web). That’s why the post treats false flags as a clinical cost and not just an operational one.

How are the “generated PA answers” being generated in the first place?

From the post: Not described. The post only calls them “generated PA answers” and says the platform “manages PA submission end-to-end.”

From web search: Forus’s site says providers write the prescription in the EHR and “Forus automatically generates and submits prior authorization forms.” Fierce Healthcare reports that each prescription gets its own AI agent that reasons over the patient’s medical history, insurance and finances, “drawing on clinical models, specialized sub-agents and what Forus has learned from millions of cases.”

Inference: The generator is most likely an LLM agent. It pulls the patient’s record through an EHR integration, retrieves evidence for each payer form question, and writes a free-text answer or picks a multiple-choice option. The judge’s retrieve-then-judge design mirrors this. That raises a question the post doesn’t answer: if the generator and judge share retrieval components, they may also share blind spots, such as a note that neither of them retrieves.

“At Forus, we developed an LLM judge to verify generated PA answers”

This is different from standard LLM judges that are used for evaluation, right? More akin to LLM verifier

From the post: Yes. The judge runs on generated PAs to “verify generated PA answers against patient clinical records and specialty-specific guidance,” and its flags send cases to humans (“every flagged case requires human intervention”). That makes it a production gate, not an offline quality score.

Inference: It differs from a typical evaluation judge in three ways:

• When it runs: online, on every case before submission, rather than offline on a test set.
• What it checks against: retrieved patient evidence (groundedness), rather than a reference answer or a quality rubric.
• What its output does: it triggers a routing decision, so a false flag has a direct operational cost. An evaluation judge’s errors only distort a metric.

“Verifier” is a fair term. In the LLM literature, though, that word often means a model that scores reasoning steps or candidate solutions, for example to pick the best of N samples. “Groundedness checker” or “guardrail” is more precise here. This framing also explains why the post evaluates the judge the way you would evaluate a classifier.

Is verification done question by question or in bulk?

From the post: Verdicts are produced per answer. Each answer is decomposed into claims, the claims are judged independently, and “question-level aggregation” rolls them up into an answer-level result. In the Crohn’s example, though, the judge flags an unperturbed answer because it conflicts with another answer, so the judge must be able to see the other answers on the form.

Inference: The unit of verification is one question’s answer, with the rest of the form available as context. The post doesn’t say whether the calls run one at a time, in parallel, or as one form-level prompt. My guess is separate per-answer calls (which can run in parallel), with the form’s context included during decomposition, since the post says decomposition “has to draw on question context, not just the answer.”

“Our system faced a calibration problem”

If they didn’t have a dataset initially, how did they know this for? Or did they assume it was the case given it’s a problem generally?

From the post: The premise is partly off, because they did have a dataset. They had already built “a rigorous gold standard dataset for our PA evals” of correct answers. Running the judge over correct answers and counting flags gives the false positive rate directly, and that is exactly the clean run. What they lacked was examples of wrong answers on the same patients, which is what perturbation adds. The post also points to industry findings [1] that judges lean conservative.

Inference: The post doesn’t say how they first noticed. The most likely route is that human reviewers in production kept clearing flags on answers that turned out to be fine, or that an early clean-run measurement showed a high flag rate. Note that “calibration” here means the judge flags too aggressively relative to the true error rate. It isn’t probability calibration in the statistical sense, because the judge outputs discrete verdicts, not probabilities.

“Emerging work like MedAgentBench [5] is more targeted, and evaluates LLM agents in simulated EHR environments. Still, no standardized benchmark exists for PA question answering.”

What's the difference between these two use cases?

From web search: MedAgentBench (Stanford, NEJM AI 2025) has 300 physician-written tasks across 100 de-identified patient profiles in a simulated EHR that follows the FHIR data standard. Tasks are either queries (retrieve information) or actions (for example, order a medication), and models are scored on task success rate.

Inference: The two differ in three ways:

• Output: MedAgentBench asks for one fact or one action with a single correct result. PA question answering asks for answers to payer questions (diagnosis, prior therapies, severity) that must be justified by chart evidence.
• Data: MedAgentBench draws on structured records (labs, vitals, procedures, diagnoses, orders). PA answers often rest on unstructured progress notes, biopsy reports and outside records.
• Grading: MedAgentBench checks whether the task succeeded. PA answers need every claim grounded in the record, and they must match payer-specific criteria.

In short, MedAgentBench tests whether an agent can operate an EHR, while PA question answering tests whether it can build a defensible case from one.

“rigorous gold standard dataset for our PA evals through a multi-annotator consensus workflow followed by expert clinical review”

Are the annotators different from the expert clinical reviewers? Why? Cost?

Inference: This is a standard tiered labeling design, and each tier handles a different kind of error:

• Random error: several annotators label each item, and taking the consensus removes individual slips. Trained non-physician staff, such as nurses or PA specialists, can do this at lower cost and higher volume.
• Systematic error and hard cases: experts (physicians or pharmacists) review the disagreements and a sample of the agreed labels. This catches mistakes that all the annotators make together.

So yes, cost is a large part of it, because expert time goes only to the cases that need it. Expert adjudication would be needed anyway, since even clinicians often disagree on this task. VeriFact reported that the best agreement between any two clinicians was 88.5% (web).

If an answer is flagged for review, what happens next exactly?

Inference: A plausible flow is:

• The flagged answer goes to a human review queue, together with the failing claim and the evidence the judge cited.
• The reviewer checks the chart and either confirms or overturns the flag.
• If the flag is confirmed, the reviewer corrects or regenerates the answer, and asks the provider for documentation if the chart is missing something.
• The corrected form is re-verified and submitted.

Claim-level verdicts make review faster, because the reviewer checks one failing claim rather than re-verifying the whole answer. The reviewers’ decisions are probably logged, and those logs are a natural source of the “confirmed errors” the post wants to mine (see Q25).

“each answer is decomposed into atomic claims”

Is this done by an LLM? If so, is it the same kind of LLM as the generation model?

From the post: It isn’t stated outright, but it’s strongly implied. The post notes that one claim (“Humira was the most recent biologic therapy”) “appears nowhere in the answer” and is implied by the question, so “decomposition has to draw on question context.” Rule-based sentence splitting can’t produce a claim like that. The method is also adapted from VeriFact and FActScore.

From web search: Both of those use an LLM for decomposition. VeriFact used Llama 3.1 70B to extract “simple sentences that encapsulate Subject-Object-Predicate relationships.”

Inference: It is almost certainly an LLM. The post doesn’t say which model, or whether it matches the generator. Decomposition is a fairly mechanical task, so a smaller and cheaper model would be a reasonable choice, with the strongest model saved for the judgment step.

Is the judge model the same as the generation model?

From the post: Not stated.

Inference: There are two reasons to prefer a different model, ideally from a different family. There’s no evidence either way about what they did.

• Self-preference: LLM evaluators tend to rate their own outputs more favorably (Panickssery et al., 2024, “LLM Evaluators Recognize and Favor Their Own Generations”; background knowledge).
• Correlated blind spots: a model that misreads a chart pattern while generating may misread it the same way while verifying.

One point cuts the other way. Their main problem is over-flagging, not leniency, so self-preference doesn’t seem to be their main failure. Their remark that “using another LLM to judge the judge is circular” shows they are aware of correlated errors in general. The same independence concern applies to the perturbation model (see Q21).

“We expand each claim into multiple semantic search queries”

How is the expansion done? SPLADE?

From the post: The method isn’t named. The example queries are full natural-language paraphrases, such as “symptoms well controlled on Humira” and “stable disease on anti-TNF therapy.” They vary clinical synonyms, brand and generic drug names, and drug class.

Inference: This looks like LLM-generated query rewriting (often called multi-query retrieval), not SPLADE. SPLADE is a learned sparse encoder. It turns one text into a single sparse vector of weighted vocabulary terms, including related terms that weren’t in the text. It doesn’t produce several readable query sentences, so you wouldn’t describe its output as “multiple semantic search queries.” The results of the separate queries are then presumably merged, for example by taking the union and removing duplicates, or by reciprocal rank fusion. The post doesn’t say which. Phrasing several queries around the same claim also makes it more likely that contradicting notes get retrieved, like the April 2026 flare note in their example.

Is their search only semantic or includes lexical as well? Why?

From the post: Only “semantic search queries” are mentioned. Lexical search isn’t.

Inference: We can’t tell, but a hybrid design would make sense, because each method covers a weakness of the other:

• Semantic (dense) search handles the synonym problem the post emphasizes, such as “eczema” versus “atopic dermatitis.”
• Lexical search (for example BM25) handles exact tokens that dense embeddings represent poorly. These include ICD codes (K50.90), lab values (CRP 12 mg/L), doses (90 mg every 8 weeks), dates and rare drug names, and PA questions depend heavily on exactly these.

VeriFact, which their pipeline is adapted from, compared dense search, hybrid dense-plus-sparse search, and hybrid search with a cross-encoder reranker (web). Two other points apply. Structured data such as medication lists and labs may be fetched by direct lookup rather than search. And the LLM query expansion partly substitutes for lexical recall, because it writes both brand and generic names into the queries.

Any domain adaptation for semantic expansion?

From the post: Nothing beyond the examples. They show clinical synonyms (“eczema” for “atopic dermatitis,” “failed therapy” for “inadequate response”) and drug-name variants (Humira, adalimumab, anti-TNF therapy).

Inference: Domain knowledge could enter at three points, none of them confirmed:

• Query generation: an LLM prompted with clinical instructions and examples. This is the cheapest option and the most likely.
• Vocabulary normalization: mapping terms to medical ontologies, such as RxNorm for drugs and SNOMED CT or ICD-10 for diagnoses, so that “Humira” and “adalimumab” resolve to the same concept.
• Embeddings: a biomedical retrieval model such as MedCPT, or a general embedding model fine-tuned on their own claim-to-evidence pairs, which their gold standard data could supply.

My guess is that prompting does most of the work, possibly with an ontology layer for drug names.

What granularity/chunking strategy for retrieved evidence?

From the post: Not stated. The diagrams show evidence as short excerpts labeled with source type and date, such as “Progress note · Apr 2026: ‘flare; symptoms recurring’,” “Lab results · Apr 2026” and “Medication list.”

Inference: That suggests sentence- or passage-sized chunks that carry metadata (document type and date). Structured data such as labs and medications is probably stored as records rather than text. Dates matter because many PA claims are about time, such as “most recent biologic” or “sustained remission.” The judge can only see that an April 2026 flare contradicts a January 2026 remission note if each chunk carries its date.

From web search: VeriFact, their reference design, split notes into 128-token chunks along semantic boundaries, then broke those into sentences or atomic propositions. It retrieved the top N facts per claim. It tested N up to 50, and performance was still improving at 50.

“Not Supported: the claim conflicts with the evidence, either through direct contradiction or through a significant omission or fabrication.”

How is the omission part different from Not Addressed?

From the post: Not Addressed means “the evidence is silent on the question.” The judge’s rationale for the adherence example, which appears when you click the verdict, draws the line: silence “isn’t fabrication either, since the claim asserts no specific finding the chart should contain.”

Inference: The difference is whether the record says anything on the point:

• Omission: the record does contain relevant information, and the answer leaves out the part that changes its meaning. For example, the answer says “methotrexate was tried” when the record says it was stopped for intolerance. That matters for a “tried and failed” question.
• Not Addressed: the record contains nothing on the point, so there is nothing to omit.

Fabrication sits between them. The claim asserts a specific finding, such as a lab value, that the chart would contain if it were true, and the chart doesn’t contain it. This boundary is hard even for people. In VeriFact, clinicians agreed only about a third of the time on Not Supported labels, and the LLM judge leaned toward Not Supported over Not Addressed (web).

“Worst-verdict aggregation means the one Not Supported claim flags the whole answer. But the Not Addressed claim does not”

So then what happens if there is a Not Addressed claim? Shouldn’t that still go to additional review?

From the post: A Not Addressed claim doesn’t flag the answer, and “that’s the distinction that keeps the judge from over-flagging.” Their reasoning is that records are routinely incomplete, so treating silence as Not Supported “would punish every incomplete chart.” The post doesn’t say what happens to these claims afterward.

Inference: Your concern is valid. The answer goes to the payer containing a claim nobody can document, and payers can deny or audit based on documentation. A middle path is to route these claims by how much they matter:

• Claims the payer’s criteria depend on, such as prior therapies or severity scores, go to a lighter “documentation gap” queue, or the provider is asked for the documentation.
• Incidental claims, such as adherence when the question didn’t ask about it, pass but get logged.

There’s also a separate question: why is the generator asserting things the chart doesn’t mention? The share of Not Addressed claims is worth tracking as a signal of generator quality, even if it never triggers a flag.

“the perturbed run measures the detection rate”

Detection rate = true negative rate?

Inference: It depends on which class you call positive:

• If a wrong answer is the positive class and a flag is a positive prediction, which is how the post frames it, then detection rate is the true positive rate for errors (also called recall or sensitivity).
• If you call a correct answer the positive class, detection rate becomes the true negative rate (specificity).

Both readings are internally consistent. The post uses the first, which is the standard one for error detection; under that framing, “false positive” means a correct answer that was flagged. Two details about the denominator: accidentally correct perturbations are excluded from it, and only direct detections count, with consistency-based flags tracked separately.

“because detection rate is only meaningful once we know the judge isn't excessively flagging”

Why?

Inference: Detection rate alone rewards flagging more. A judge that flags every answer has a 100% detection rate. A judge that flags answers at random with probability p has a detection rate of p, with no skill at all. So detection rate only shows skill as the gap above the false positive rate. For example, 60% detection with a 40% false positive rate means the judge barely tells wrong answers from right ones, while 60% detection with a 2% false positive rate is a useful judge. The two numbers have to be read as a pair, like recall and precision.

There’s a second reason. With a high false positive rate, some flags on perturbed answers may be for reasons unrelated to the perturbation, so the detection count overstates real catches. In principle, their claim-level rationales could be used to check that each flag points at the perturbed claim. Anchoring on the false positive rate also reflects their priority, since over-flagging is the failure they set out to fix.

“One perturbation, two flags, and the second one is, interestingly, a behavior we hadn't explicitly designed the judge to produce.”

I don’t fully understand the difference between these two flags.

From the post: The interactive demo labels the two flags explicitly:

• Flag 1, “direct detection,” is on Q1, the answer they changed. The diagnosis now says ulcerative colitis, but the chart documents Crohn’s disease, so the answer contradicts the record.
• Flag 2, “consistency-based,” is on Q2, an answer they didn’t change. “Stricturing disease with perianal fistulas; transmural inflammation” is still true according to the chart. But those features are hallmarks of Crohn’s, not ulcerative colitis. The demo’s rationale says Q2 “no longer fits the diagnosis it’s meant to support, so the judge flags the mismatch.” Perturbing Q2 instead produces the mirror case: Q1 gets flagged “even though this answer is chart-supported.”

Inference: Flag 1 checks an answer against the record, and flag 2 checks an answer against another answer. Measured against the chart alone, flag 2 is a false positive. It’s a useful one, though, because the submitted form as a whole would no longer make sense. That’s why the post counts the two kinds separately.

“Our approach was to build a perturbation model, seeded with structured context from the patient's record, so its wrong answers stay anchored to the patient's actual medical history. For free-text questions, the model generates a plausible wrong answer.”

So they’re just giving the model relevant context and asking it to generate answers that seem plausible but are wrong? Any configurations to think about beyond the context and the prompt?

From the post: Essentially yes. The model gets structured context from the record. For free-text questions it writes a plausible wrong answer. For multiple-choice questions it picks a different option that is “clinically coherent in context but factually unsupported.” The constraints are definitive assertions, specific and falsifiable claims, no attestations, and about 30% of questions perturbed per form.

Inference: Other settings worth deciding, grouped by stage:

• Before generation: which questions to perturb (at random, or stratified by question type), and which error type and level of subtlety to aim for. Ideally these come from an error taxonomy (see Q26).
• During generation: which model to use (ideally from a different family than the judge, to avoid shared blind spots), and how many candidates to sample per question.
• After generation: validity checks (is the new answer actually contradicted by the chart, and is it falsifiable?), plus logging so that runs can be reproduced.
• Across the dataset: splitting the train and test sets by patient rather than by question, so prompt tuning never sees a held-out patient’s record.

“Three ways a synthetic perturbation can fail to be a useful test”

Once they observed these, what did they do to avoid them? Prompt adjustments?

From the post: Each failure mode got a different fix, applied at a different stage:

• Accidentally correct (for example, they swapped in azathioprine, but the patient really did try it): “Detect & exclude from metrics,” which is a check after generation.
• Unfalsifiable (for example, “all alternative therapies were tried”): “Require specific, falsifiable claims,” which is a constraint on generation.
• Administrative noise (attestations): “Filter out before perturbing,” which limits which questions are eligible.

Inference: The post doesn’t say how these were implemented. The likely mix is prompt instructions for falsifiability, a question-type filter for attestations, and some kind of validator for accidentally correct cases. That validator has a trap. If it relies on the judge’s own “Supported” verdict to decide that a perturbation was accidentally correct, it will also throw out real misses and inflate the detection rate. The check has to be independent of the judge, for example a lookup in the structured medication history, or human review of every excluded case.

“In production, real errors often cascade through related questions”

What does this mean?

Inference: A PA form asks many related questions about one patient, and the generator answers all of them from the same reading of the chart. If that reading is wrong at one point, the error spreads to every answer that depends on it. For example, if the generator takes the diagnosis to be ulcerative colitis, it may also list therapies relevant to that disease, report a different severity score, and argue medical necessity for the wrong condition. So real errors are correlated across questions, while their perturbations change one answer at a time.

This matters for the consistency signal. In a full cascade, all the answers could be wrong in a way that is consistent with each other. The cross-answer check then wouldn’t fire, and only checking against the chart would catch the error. Single-answer perturbations may therefore make consistency-based detection look more useful than it really is.

“the judge flags a different, unperturbed answer because it conflicts with the perturbed one”

Is their judge actually looking for inconsistency across questions or is this a hypothetical?

From the post: It’s described as observed behavior, not a hypothetical: “The judge flags two things,” and the second is “a behavior we hadn’t explicitly designed the judge to produce.” They also say they have to separate direct detection from consistency-based detection in their evaluation, which would only be necessary if it actually happens in their runs. The sentence you quoted is how they define that second category. They don’t report how often it happens, though, and the interactive demo is scripted.

Inference: Since they didn’t design it, the judge isn’t explicitly checking for consistency across questions; the behavior comes from somewhere else. A plausible mechanism is decomposition. For Q2 (“What clinical features support the diagnosis?”), decomposition that draws on question context would produce an implied claim like “these features support a diagnosis of ulcerative colitis,” taking the diagnosis from Q1’s answer. The chart contradicts that claim, so Q2 gets flagged. If that’s right, consistency detection only works when one question explicitly refers to another answer.

“One is mining production data for confirmed errors”

How is an error confirmed?

Inference: Possible sources of confirmation, from most to least direct:

• Reviewer outcome: a human checks a flagged answer against the chart and corrects it.
• Provider correction: the prescribing office amends an answer after seeing the form.
• Payer response: a denial or a request for more information that points at a specific answer. Denial reasons are often vague, so linking them to an answer takes manual work.
• Audit sampling: running the gold-standard annotation process on random production cases, including ones the judge passed.

The first three mostly surface errors someone already noticed, and the reviewer route only confirms errors that the judge itself flagged. Errors the judge misses won’t show up unless something downstream catches them. So mining confirmed errors under-represents exactly the misses the evaluation most needs, and random audits are the way to correct for that.

“Another is building an error taxonomy from real cases so that perturbations cover the actual distribution of mistakes rather than the ones a model finds easy to invent”

How is this different from mining production data in the previous sentence?

Inference: They work at different levels:

• Mining gives you individual real errors. These can seed the perturbation model directly, for example as few-shot examples, which makes each perturbation look more like a real mistake.
• A taxonomy classifies those errors and records how often each type occurs. Example types include a wrong date range, an outdated lab value, past history stated as current, a wrong drug in the prior-therapy list, or a flipped negation. The taxonomy tells the generator how many of each type to produce, so the test set matches the real mix. Otherwise it would over-represent errors that are easy for a model to invent, such as swapping a diagnosis.

Mining makes each example more realistic, and the taxonomy controls which kinds of errors appear and in what proportions. A taxonomy also lets you report detection rate per error type, which shows where the judge is weak. You build the taxonomy from mined errors, so mining comes first.

“where the perturbation model is trained to produce errors the judge misses”

How would you actually do this? Prompt iteration, fine-tuning or something else?

Inference: Three options, from cheapest to most involved:

• Search without training: generate many candidates, keep the ones the judge misses, and feed them back as few-shot examples in the next round. No model weights change.
• Preference fine-tuning (for example DPO): train on pairs in which the missed perturbation is preferred over the caught one.
• Reinforcement learning: reward the generator whenever the judge misses its error.

Every option needs a validity check that doesn’t depend on the judge. Otherwise the generator learns the easiest way to beat the judge, which is to produce accidentally correct or unfalsifiable perturbations, the same failure modes the post documented. A perturbation should count as a success only if it meets three conditions. It must be verifiably wrong against the chart (checked by structured lookup, a separate checker that sees the gold answer, or human spot checks). It must be clinically plausible. And the judge must miss it. Restricting generation to the error types in the taxonomy (Q26) also keeps the errors realistic.

What could “production monitoring” actually look like for their use-case?

Inference: Grouped by what each signal tells you:

• Over-flagging: the share of flags that reviewers overturn, broken down by drug, payer and question type.
• Missed errors: payer denials or information requests on answers the judge passed, plus random audits of unflagged answers.
• Drift: changes over time in the flag rate and in the mix of verdicts. A rising share of Not Addressed claims, for example, can mean that retrieval broke or that a new EHR integration is sending incomplete records.
• Retrieval health: the share of claims for which nothing was retrieved.
• Patient impact: time from prescription to submission for flagged versus unflagged cases, which shows what over-flagging actually costs.

A useful addition is canary testing: regularly running known perturbed forms through the live pipeline to confirm that detection hasn’t dropped after a model or prompt change.

How would you improve this system if you were building it today?

Grouped by component:

• The judge: make the cross-question consistency check a deliberate step instead of a side effect, so it can be measured and can catch full cascades (Q23). Route claims that payer criteria depend on separately, so a Not Addressed verdict on one of them goes to a documentation queue (Q17).
• The flagging threshold: have the judge output a confidence for each claim, and set the threshold from the relative cost of a missed error versus a false flag, instead of using a fixed worst-verdict rule.
• Perturbations: generate them from an error taxonomy (Q26), include errors that cascade across several answers, and check for accidentally correct cases independently of the judge (Q22).
• Evaluation reporting: report detection rate by error type with confidence intervals. Also keep a small held-out set of real confirmed production errors as the final test, since synthetic detection rates probably overstate performance on real errors.
• Monitoring: add random audits of unflagged answers, which are the only unbiased way to estimate how many errors the judge misses in production (Q25).

“In the case of AI agents, Cresta suggests how to modify the agent’s prompt or other parts of its configuration and then runs tests to validate that the agent’s performance improved as a result.“

Does “Cresta” here refer to a Cresta employee or a Cresta system? In other words, is this done manually or automatically?

[Source] The post doesn't say. The sentence is built as a parallel to the preceding one about human agents, where Cresta the platform delivers hints and coaching, so the grammar points toward the system.

[Inference] The two verbs are doing different amounts of work. "Runs tests" is plainly automated — that's the evaluator toolkit and simulation suite described later. "Suggests how to modify" is the softer claim, and most likely means the platform surfaces a diagnosis (this turn failed this behavior, here's why) while a person decides the actual edit. The post's own Lessons Learned section supports that reading, since its entire argument is that humans stay in the loop on targeted decisions. So the honest summary is automated detection and automated validation, with a human-authored change in between. "Suggests" is the kind of verb that lets a vendor describe either arrangement.

Are Cresta’s agents primarily text-based or voice?

[Source] This post is modality-agnostic. It refers to "chat and call transcripts" and mentions both inbound and outbound conversations.

[Search] Voice is clearly the emphasis across the rest of their writing. Their Voice Virtual Agent announcement says voice is "the top used channel at over 70% of interactions" and criticizes competitors for "focusing exclusively on chat," and their latency engineering post is entirely about the voice pipeline — speech recognition, voice activity detection, speech synthesis.

[Inference] Both channels are supported, but the engineering problems they choose to write about are voice problems. The "interruptions" metric listed in this post only makes sense for voice at all. I'd read the post as voice-first with chat along for the ride.

“Cresta fixes this cold start problem by grounding the design in historical data”

Doesn’t cold start imply no historical data?

[Inference] You're right that the term is being stretched. Cold start conventionally means having no data about a new user or item. Here the cold start belongs to the agent, not the enterprise: the new AI agent has no interaction history of its own, while the enterprise has millions of transcripts. So what they're describing is a transfer-learning move — bootstrap the new agent from the behavior of the incumbent human agents. That's a reasonable idea, but calling it "fixing cold start" is marketing language; a stricter description would be a warm start from a related distribution. And the distributions genuinely differ: customers ask different things of a bot than of a person, and they behave differently once they know which one they're talking to.

What would you do if you actually didn’t have any historical data and wanted to launch something like this?

[Inference] Roughly in order of cost, I'd do five things. First, mine the written artifacts that exist even without transcripts — standard operating procedures, help-center articles, canned agent responses, past ticket subject lines, and abandoned queries typed into a chat widget. Those give you intents without needing dialogue. Second, have subject-matter experts record twenty to fifty think-aloud role-plays, which becomes the seed corpus your personas are drawn from. Third, generate synthetic conversations from the procedure documents, but validate them against the role-plays so you aren't only testing the agent against your own assumptions. Fourth, launch deliberately narrow, or run the agent in shadow mode alongside human agents so it produces responses that get scored but never sent. Fifth, instrument heavily, because the first few thousand real conversations are the actual asset and you want the relabeling loop to be cheap from day one.

The thing to watch is that the first three all encode the same prior. Only real traffic breaks it.

“finds the major clusters of customer issues”

How?

[Search] Their Insights announcement is more specific: Topic Discovery uses "unsupervised LLM clustering on conversation reasons," and generates "an exhaustive taxonomy and up to three levels of subcategories" without teams having to predefine categories.

[Inference] In practice that reads as a two-stage pipeline. An LLM first extracts a short conversation reason from each transcript, compressing a long call into a normalized phrase. Clustering then runs over those compressed reasons rather than over raw transcripts — most likely embeddings plus a clustering algorithm, with an LLM naming clusters and merging them hierarchically into the taxonomy. Clustering on extracted reasons instead of full text is the important design decision, because small talk, greetings, and hold messages would otherwise dominate the similarity signal and you'd end up clustering conversations by style rather than by problem.

Why do so many of their cluster examples sound very similar to each other?

[Inference] Two explanations, and they have different implications. If the screenshot is showing sibling leaves beneath one parent node, then similarity is expected and fine — "billing, unexpected charge" and "billing, duplicate charge" are supposed to differ by one discriminating detail, and the three-level taxonomy makes that likely. The less comfortable explanation is under-merging, which is a real failure mode of LLM-driven taxonomy induction. The model finds distinctions that are linguistically genuine but operationally meaningless, because nothing in the clustering objective penalizes redundancy. The test I'd apply: if two clusters would route to the same conversation flow and trigger the same tool calls, the taxonomy is over-split for the purpose it's being used for. Your suspicion is worth holding onto.

“how human agents resolved them”

Do we know for sure that the human agents' approaches were correct? How?

[Inference] "Proven" is doing unearned work there. Transcripts record what humans did, not what was correct. There are three ways to get closer to correctness, none of them free. You can filter by outcome rather than by behavior, keeping only conversations with a good resolution signal such as no callback within some window, positive satisfaction score, or no subsequent refund reversal — which turns plain imitation into outcome-weighted imitation. You can stratify by agent and learn only from the top quartile on the relevant metric, which is the standard contact-center move and something Cresta's quality management product already scores. Or you can treat the mined flow as a hypothesis rather than ground truth and put it in front of subject-matter experts before it becomes the spec. There's also a survivorship risk worth naming: shortcuts agents use to hit handle-time targets look indistinguishable from best practice in a transcript.

“Insights surfaces real customer phrasings and look-alike intents”

What does this mean?

[Inference] They're two separate things. Real customer phrasings means the utterances people actually produce, as opposed to how the business names the intent internally — customers say "I got charged twice," while the internal taxonomy says "duplicate transaction dispute," and it's the former you need in the router's examples. Look-alike intents means pairs that sit close together in surface form but need completely different handling. "I want to cancel my order" and "I want to cancel my subscription" share most of their words and then diverge into different flows touching different systems. Surfacing those pairs during scoping tells you in advance where your router will confuse itself, so you can add a disambiguation turn or supply discriminating examples before the confusion shows up in production traffic.

“we can explicitly define what the AI agent will handle and what falls out of scope”

What are realistic examples of things that would fall out of scope?

[Inference] Exclusions tend to fall into four groups. There's regulated or liability-bearing work — debt collection promises, insurance coverage determinations, medical or legal advice, anything that creates a binding commitment on the company's behalf. There's high blast radius — refunds above a threshold, account closures, changing a payment method, anything hard to reverse. There's emotionally loaded contact — bereavement, fraud victims, threats of self-harm, and retention saves where human judgment carries the conversation. And there's the long tail, meaning intents that appear a handful of times a month where the cost of building and maintaining a flow exceeds the containment you'd gain. By raw count the long tail is the largest category. The first two are the ones that get written into the contract.

How is “automation readiness” determined?

[Source] This post lists it as something Insights quantifies, with no method given.

[Search] Their dedicated Automation Discovery post is more forthcoming: "The score is driven by structural and operational signals observed in the data, including the frequency and nature of deviations, as well as the level of inferred integration complexity."

[Inference] Unpacking those two signals, deviation frequency is a proxy for how well-structured an intent is — if most conversations follow one dominant path, a state machine can capture it, and if they scatter, the intent needs judgment. Integration complexity is a proxy for build cost, meaning how many backend systems the flow has to touch. So readiness is roughly path regularity divided by integration cost, weighted by volume. Worth noticing that this is a build-feasibility score, not a risk score. Nothing in it speaks to whether automating a given intent is safe, and the post's framing quietly blends the two axes.

“Automation Discovery reconstructs representative dialogue flows from the transcripts”

If this tool is looking at a very large number of transcripts (presumably not fitting in a single context window), how is it generating the desired information correctly?

[Source] Not addressed here. Their Automation Discovery post says only that they use "LLMs together with deterministic analysis."

[Inference] Nobody puts a million transcripts into one context, so the shape is almost certainly a four-stage pipeline. First, per-transcript extraction, running one model call per conversation in parallel, compressing each into a structured trace: the reason, an ordered list of steps, tool-like events, and the outcome. This is the only stage that sees raw text and it's parallel. Second, cluster the traces rather than the text. Third, align the step sequences within each cluster, which is closer to process mining than to language modeling and can be done deterministically with frequent-sequence mining or transition counting. Fourth, bring an LLM back in only at the end, to name states and write descriptions over an aggregate that now fits comfortably in context. Their phrase "deterministic analysis" most plausibly points at the third stage. The quality risk sits in the first: whatever schema the extractor uses silently bounds what the final state machine is capable of expressing.

“State machine skeleton: It identifies the dominant paths customers take to reach resolution”

Per intent, or more general?

[Inference] Per intent, almost certainly. A single global state machine spanning all customer issues would be close to useless, because the states for rescheduling an appointment and disputing a charge share nothing except authentication. The pipeline ordering in the post supports this: Insights clusters issues first, then Automation Discovery reconstructs flows, so the natural unit is one flow per cluster, with shared preamble states like greeting and identity verification and shared exit states like escalation factored out as common subgraphs. Their decentralized-architecture post is consistent with this, since it names sub-agents for "authentication, policy verification, troubleshooting workflows, payment handling" — which is exactly a shared-preamble-plus-per-intent-body decomposition.

Is a human the primary consumer of the output of the Automation Discovery tool to guide the development of the agent? Or is it being directly fed into a system somewhere?

[Search] Both, according to their Automation Discovery post. It supports human inspection, and it "can export a structured draft prompt derived from the workflow, serving as a scaffold for AI agents."

[Inference] "Draft" and "scaffold" are the operative words — this is a human-reviewed handoff, not a closed loop. That's the right call given what the artifact is. The mined flow encodes whatever the human agents happened to do, so pushing it straight into production would automate their mistakes at scale. The post you're reading agrees implicitly by calling the scoping output a "v1 spec," which presupposes somebody reviews it.

“Tool catalog: identifies critical data dependencies and system touchpoints”

How can you get this information just from transcripts?

[Search] Their Automation Discovery post answers this directly: "When agents say things like 'let me check your order,' 'I'll submit a claim,' or 'I need to verify eligibility,' those patterns often indicate interaction with backend systems."

[Inference] So it's linguistic evidence that a system call happened, not observation of the call itself. Two further signals are usually available and they don't mention either: long silences or explicit hold events in a call mark a lookup, and the shape of the data the agent reads back — an order number, a balance, a ship date — reveals both which system was queried and roughly what it returned. The limitation is real, though. This gives you a hypothesis list of integrations with no guarantee of completeness, no API contract, and no visibility into any system the agent used without narrating it. Screen recordings or CRM audit logs would be the ground truth; transcripts are the cheap proxy for it.

How is “Tool use and data needs” different from “Tool catalog”?

[Source] The post lists them as separate bullets and the distinction it draws is thin. Tool catalog identifies "critical data dependencies and system touchpoints" and enables function schema specification; tool use and data needs catalogues required internal systems and data such as CRM lookups and order adjustments, and informs function call specifications.

[Inference] As written these overlap heavily, which by a MECE standard is a defect in the post rather than a distinction you're missing. The most charitable reading is that the catalog is the inventory — which systems exist and get touched — while tool use and data needs is the contract, meaning what each call requires as arguments and what it returns. Inventory versus signature.

“that might merit separate specialized agents”

Is the appeal of separate agents a clean context window or different capabilities (whether through a different model, harness or skills)?

[Source] This post doesn't say.

[Search] Their decentralized-architecture post makes the case, and the reasons it gives are mostly neither of your two options. It cites avoiding a single point of failure, since if a central orchestrator "falters due to an unexpected scenario, a model update, or a spike in call volume, the entire experience may degrade"; avoiding errors that "accumulate across long conversations"; and latency, because "every decision and turn reroutes through the orchestrator." It does also describe keeping "the AI focused on the immediate step," which is your clean-context argument.

[Inference] In practice I think prompt focus is the dominant real benefit, so your first option. A sub-agent's prompt contains only the rules for its own task, which raises instruction-following accuracy and cuts token cost. Independent testability is second, and this post names it outright — "each skill can be built and tested independently." Running different models per sub-agent is easy to do and occasionally worth it, a small fast model for authentication and a stronger one for troubleshooting, but it isn't the headline reason.

“and a world state the agent must navigate: authentication, entitlements, status, partial data, the messy stuff”

Why is this in the simulated visitor spec as opposed to the test spec? Does their version of visitor both the simulated user and the task?

[Source] Yes, it bundles them. The simulated visitor spec as described carries four things: a persona with tone and vocabulary, a goal defining "what 'done' means," the world state, and behavior policies covering how the person asks, clarifies, and pushes back. Separately the post says "we know 'what should happen' in each scenario," so the expected outcome travels with the scenario too.

[Inference] Your instinct that these are separable is right, and separating them would be the better design. Persona, world state, and goal form a cross-product you'd want to sample along independently, so you could hold a persona fixed while varying account state, or the reverse. Bundling them into a single artifact means every combination is a hand-built object and your coverage is limited to whatever someone thought to write down. The likely reason it's bundled is how they're generated: if each visitor is derived from one real conversation, the persona and the world state come from the same source record and arrive already entangled.

“Because these simulated visitors are generated from de-identified real conversations”

How generated?

[Source] No method given beyond the de-identification.

[Inference] The plausible pipeline is to take a real transcript, strip personally identifying information using named-entity redaction plus format-based scrubbing for things like card and account numbers, then have an LLM read the cleaned transcript and emit the structured spec. It would infer the persona from how the customer wrote or spoke, the goal from what they were trying to achieve, the world state from facts established during the conversation — were they authenticated, did they have an active policy, was the order partially shipped — and the behavior policies from observed conduct such as pushing back, wandering off-topic, or misunderstanding a rule. Then you discard the original transcript and keep only the spec, which is what makes the artifact privacy-safe: it's a description of a customer rather than a copy of one. Whether they check the generated spec back against the original for fidelity is unstated, and that's the thing I'd most want to know.

“simulate full multi-turn conversations between each virtual customer and the AI agent”

1. How to sandbox to ensure no cross-context contamination?
2. Different models or same? Does it matter?

[Inference] (a) The contamination risks are distinct and each needs its own control. Within a single run, the visitor model must not see the agent's system prompt, tools, or internal state — only its own spec plus the dialogue so far — which means two separate processes with separate contexts, not two personas sharing one context. Across runs, you need no shared memory, fresh conversation identifiers, and a reset of the mock backend between runs, because otherwise run N's writes, a payment recorded or a ticket opened, silently change run N+1's world state. The agent's own tools have to hit mocks or a seeded test database rather than production, and this is the one that catches people out, since a tool that looks read-only often still writes an audit record. Finally the grader shouldn't know which configuration produced a transcript, or you introduce scoring bias precisely when you're diffing prompts against each other.

(b) Different models, and yes it matters. Using the same model on both sides is the classic evaluation trap: the visitor and the agent share blind spots and failure modes, so the agent scores better than it deserves and the visitor never generates the confusions a different model — or a real person — would produce. Running a different family for the visitor, and ideally a third for the judge, buys you independence. The cost is that your simulated population now inherits whatever quirks the visitor model has, which is exactly why the pool still needs to be grounded in real transcripts.

“The result of each simulation run is a scored transcript with precise reasons for any failures and artifacts users can diff across prompts, tools, or models.”

Same scoring methodology as online and human agents?

[Source] Partly, and the post is explicit about the shared part. It says "a lot of AI agent requirements map exactly to behaviors and are the same for human and AI agents," and that customers running both "can often share the same behavior definitions between the two." That's about Opera behavior definitions applied in production.

[Inference] But simulation scoring has something production scoring structurally cannot have, which is a known expected outcome — the post's "we know 'what should happen' in each scenario." So simulation can score outcome correctness by direct comparison, while production can only score behavior adherence by judgment, because nobody knows the counterfactual for a live call. My read is a two-layer scheme: shared behavior evaluators, the Opera definitions, applied everywhere, plus an outcome-comparison layer that exists only offline. That asymmetry is worth holding onto, because a high simulation pass rate and good production behavior scores are not measuring the same thing and can move independently.

“Because the simulator can spin up hundreds of variations of a scenario, it’s great for catching edge cases that never surfaced in a low-volume manual test.”

How to encourage diversity and exploration to uncover more edge cases?

[Inference] Naive sampling from one model collapses toward the mode, and you end up with hundreds of near-identical polite customers. The things that actually work: factorize and grid rather than sample, meaning enumerate the axes explicitly — authentication state, data completeness, emotional register, language proficiency, interruption behavior — and cover the cross-product instead of asking a model for a hundred varied customers. Seed from the tail of real data by sampling source transcripts inversely to cluster frequency, so rare intents are over-represented in the visitor pool. Generate adversarially by conditioning the visitor on a known failure mode, such as insisting on a policy exception, giving the account number with one digit wrong, or changing goal mid-conversation. Apply novelty pressure by embedding generated scenarios and rejecting any that sit too close to an existing pool member, which is cheap and surprisingly effective. And mine production for what you missed, which is what they describe doing post-launch and is the only genuine source of unknown unknowns.

The measurable version of all this is coverage over the state machine's transitions: which edges has the pool never exercised?

“after go-live, we also keep expanding the pool with new conversation patterns as real users interact with the agent”

Automatically or manually? How?

[Source] Unstated. The full sentence continues "so we can continually run regression tests."

[Inference] The mining step is almost certainly automatic, since it's the same visitor-generation pipeline from question 18 pointed at production transcripts instead of historical ones. The interesting decision is candidate selection, and the natural signals are conversations where the agent escalated, where the customer repeated themselves or expressed frustration, where an evaluator flagged a behavior violation, and where the conversation reason falls outside the existing taxonomy.

[Search] That last one is exactly what their Trends feature detects — their Insights post describes Real-Time Trends surfacing "emerging phrases, unusual spikes, and cross-cutting patterns," which is a natural feed into this.

[Inference] Promotion into the regression suite should be manual, or at minimum reviewed. An automatically added test encodes whatever the agent did as the expected behavior unless a human supplies the expected outcome, and the post's own emphasis on knowing "what should happen" implies someone has to.

“encapsulates everything the agent needs: its prompts, decision logic”

What is “decision logic”?

[Source] The post doesn't define it. It lists the term alongside prompts, tool integrations, and guardrails, and elsewhere mentions adjusting a "workflow stage" as a thing you can do to a config.

[Search] Their decentralized-architecture post is more concrete, describing "deterministic state management" that keeps "track of exactly where a customer is in a process, triggering the right actions at the right moment," combined with dynamic prompting that keeps "the AI focused on the immediate step."

[Inference] So decision logic is the non-LLM part of control flow — the state machine from the scoping phase, rendered as configuration. Which stage the conversation is in, which transitions are legal from here, which tool fires on entering a state, when control hands off to another sub-agent, and when the conversation escalates. The language model handles what gets said within a state; the decision logic constrains which states are reachable at all. That's the hybrid architecture they advertise, and it's also why a config bundle is diffable in a way a single prompt isn't: state transitions are structured data, prompt prose is not.

“Under the hood, the agent’s reasoning and capabilities are extended by Cresta’s AI Agent Framework which is a declarative framework that lets developers register backend functions”

What is “a declarative framework”?

[Inference] Declarative means you describe what something is and the framework works out how to execute it, as opposed to imperative, where you write the control flow yourself. Concretely here, a developer writes an ordinary function and annotates it with a schema — a name, a description, typed parameters, a return type — and the framework handles everything downstream: exposing it to the model as a callable tool, validating the model's arguments against the schema, invoking the function, and feeding the result back into the conversation. The developer never writes the "if the model requested this tool, parse its arguments and dispatch" plumbing. Python decorators are the usual surface syntax for this. The real payoff is that the tool's declaration and its implementation are the same artifact and therefore can't drift apart — which matters, because "schema mismatches" is one of the failure types the post names in its optimization loop.

“The framework is optimized for ultra-low latency”

How?

[Source] The post asserts it without method, tying it only to "preserving a smooth real-time conversation."

[Search] Cresta has a whole post on this. Their stated budget is that pauses beyond about 300 milliseconds "can feel unnatural" and anything past roughly 1.5 seconds "can rapidly degrade the experience." Component budgets they cite: audio preprocessing 25 to 50 ms, speech recognition 200 to 300 ms, turn detection at least 600 ms, language model time-to-first-token anywhere from 250 ms to over a second, and speech synthesis first byte 100 to 500 ms. The named techniques are streaming APIs with no DNS lookup in the critical path, reused connections to the model, WebRTC transport which they say "can reduce latency by up to 300ms," guardrail model calls issued concurrently with the main call rather than after it, speculative triggering meaning "starting the LLM call before the user fully stops speaking," hedging meaning "launching multiple LLM calls in parallel and using whichever returns first," and smaller models in the live loop since "reasoning models generally can't be used within the live response loop."

[Search] On tool calls specifically, which is what the framework sentence is really about, their sharpest observation is that if you look something up before the model can say anything at all, "the 1 LLM first-token latency becomes 1 LLM latency + 1 LLM first-token latency." Their mitigations are to fire predictable lookups concurrently at the start of the conversation, emit filler or wait messages for calls under about ten seconds, and run anything longer asynchronously.

[Inference] That last cluster is the actual content behind "optimized for ultra-low latency." The framework's job is to make a registered function's round trip overlap with generation rather than block it.

“Cresta’s framework supports this composition, so each skill can be built and tested independently”

What does skill mean here?

[Source] The post uses skill more or less interchangeably with specialized sub-agent — the preceding sentence is about "specialized sub-agents for distinct tasks." It also uses "micro-skills → agents" back in the scoping section, describing task decomposition that informs multi-agent architecture.

[Search] Their decentralized post names the actual units: "authentication, policy verification, troubleshooting workflows, payment handling, and more."

[Inference] So a skill here is a bounded task with its own prompt, its own tools, and its own entry and exit conditions — one coherent chunk of the state machine, packaged so it can be exercised in isolation. Worth noting this is not the Anthropic sense of "skill" as a loadable instruction bundle; it's much closer to a microservice with a prompt attached. A good test for whether something deserves to be its own skill is whether you can write a pass/fail condition for it without referring to the rest of the conversation.

“then unified in an orchestrated flow”

By an orchestrator agent?

[Source] The post says "orchestrated flow" and leaves the mechanism unspecified.

[Search] Their architecture post argues explicitly against a central orchestrator agent. It says a central orchestrator "can also create a single point of failure," that errors "accumulate across long conversations," and that centralized designs "introduce extra steps, since every decision and turn reroutes through the orchestrator." Their alternative is "a network of specialized subagents" that "handle their areas of expertise directly and in parallel," coordinated by deterministic state management.

[Inference] So "orchestrated" here means orchestrated by configuration, not by a supervising model. The state machine decides which skill is active and when control transfers, and there's no LLM sitting above the sub-agents making a routing decision every turn. That's a substantive architectural claim and a defensible one for voice, since it removes a model call from the critical path and makes handoffs deterministic and independently testable. The tradeoff is rigidity: anything the state machine didn't anticipate has no path through the graph and falls through to the escalation edge.

“The optimization loop is simple: Tweak the agent config or remote functions, run the test set, and fix whatever shows up in red (bad routes, schema mismatches, guardrail violation etc)”

Overfitting concerns?

[Source] Not addressed. The post presents the loop approvingly and notes that every run is archived for comparison against the last green one.

[Inference] Yes, and I'd call this the sharpest methodological gap in the post. Fixing whatever shows up red and then rerunning the same test set is gradient descent on the test set with a human as the optimizer. Four specific risks follow. You get test-set memorization, where prompt edits patch individual failing scenarios rather than the underlying behavior, which shows up as a pass rate that climbs while production quality doesn't move. Nothing in the description mentions a held-out split the config is never tuned against, which is the minimum defense. With hundreds of stochastic scenarios and repeated runs you get multiple-comparison drift, where some red-to-green flips are noise and chasing them adds prompt text that constrains nothing real. And prompt bloat is the visible symptom of all of it: each patch appends a rule, the prompt grows, instruction-following degrades, earlier rules get quietly ignored, and the fix for failure N breaks passing case M.

What I'd expect in a mature version: a frozen holdout set, freshly generated scenarios each cycle, several seeds per scenario with a significance test on the difference rather than eyeballing red versus green, and a regression gate that reruns previously-passing cases. Their post-launch mining partly addresses this by continuously refreshing the pool, which is the right instinct — but it only works if new scenarios are being added faster than they're being tuned against.

“If it clears the requirements (pass rate, no criticals, tool-accuracy targets)”

1. What is pass rate?
2. What are tool-accuracy targets and why do we care about them?

[Inference] (a) Pass rate is the fraction of simulated scenarios where the agent reached the expected outcome, with the post's "we know 'what should happen'" providing the comparison basis. It's a conversation-level metric, which their companion post distinguishes from turn-level checks. Two things need pinning down before a pass rate means anything across versions: what counts as a pass, meaning exact outcome match versus evaluator-judged acceptability, and how the scenario pool is weighted, since a pass rate computed over a pool that over-samples easy intents isn't comparable to one computed over a harder pool.

(b) Tool accuracy is whether the agent called the right backend function, at the right moment, with correctly-formed arguments. It decomposes into at least four failure modes worth tracking separately: the wrong tool was selected, the right tool was called at the wrong point in the flow, the arguments were malformed or hallucinated — the "schema mismatches" the post names — or no tool was called when one was required.

It earns its own gate for three reasons. It's where the irreversible damage lives, since a clumsy sentence is recoverable and a refund issued to the wrong account is not. It's the one dimension where the agent touches systems of record, so its errors escape the conversation and persist. And unlike conversational quality it's cheaply and objectively checkable, because you know what the correct call was, which makes it a good hard gate rather than a soft score.

“Once an AI agent is live and serving production traffic, we monitor its performance”

Who is “we” here? Cresta or the customer?

[Source] Ambiguous, and the referent shifts within the section. The sentence continues "much like we monitor human agent conversations," and human-agent monitoring is done by the customer's quality team using Cresta's tools. But the following sentences say "we monitor all the metrics we care about tracking," and then describe "making it easy for our customers to align the system," which puts customers in the third person.

[Inference] Most likely both, with the weight on the customer. Everything described — Dashboard Builder, Opera, AI Analyst — is a self-serve product the customer operates, and the entire framing of Opera is that customers define their own behaviors. Cresta presumably also watches closely during deployment and ramp. The slippage in "we" is worth noticing because it conceals an operating-model question: is post-launch tuning a service Cresta performs, or a product the customer runs? The post reads as the latter, sold with the former during onboarding.

“This includes metrics common to AI agents, or specific to certain types of AI agents. For example: Interruptions”

What are interruptions?

[Inference] An interruption is the customer speaking while the agent is still speaking — barge-in. It gets tracked because it's a cheap, automatically detectable proxy for several distinct problems at once: the agent is talking too long, the turn-detection threshold is mis-tuned so the agent starts before the customer has finished, the response was wrong and the customer is cutting in to correct it, or the customer is simply frustrated. A rising interruption rate is an early warning that requires nobody to read a transcript. Its weakness is exactly that ambiguity — it won't tell you which of those four is happening, so it's a trigger for investigation rather than a diagnosis.

“Trends and Anomalies page automatically detects shifts in the distributions of conversation reasons”

Is this mechanistically different from Insights?

[Search] Their Insights post draws the distinction directly. Topic Discovery provides "durable structure: a trusted view of what customer conversations are about," while Real-Time Trends provides "the real-time signal: visibility into what is changing right now, including events no taxonomy was built to catch."

[Inference] So they're layered rather than alternative, and the mechanical difference is the unit of work. Topic Discovery is a batch clustering job that induces a taxonomy: expensive, run periodically, output is a stable label set. Trends and Anomalies is a streaming statistical job over labels that have already been assigned: cheap, run continuously, output is a change signal on distributions you already defined. The phrase about "events no taxonomy was built to catch" implies a second component that isn't just distribution monitoring — emerging-phrase detection over unlabeled text, which catches the thing that doesn't have a bucket yet. That's the piece that genuinely overlaps with clustering, and is presumably a lighter-weight version of it run over a recent window.

“Show them a specific conversation turn where the agent did something questionable”

How do you know in advance if an agent did something questionable without showing it to a human?

[Source] The post doesn't address the selection problem, which is the load-bearing gap in its Lessons Learned section. Their companion evaluator post doesn't either — it covers how to calibrate evaluators, including a preference for binary over numeric scales because "numerical scales are inherently more subjective than binary classifiers," but not how candidates get chosen for review in the first place.

[Inference] This is the classic active-learning problem, and the key realization is that you don't need to know the right answer to select well. You need a cheap signal that correlates with being wrong. The usable ones, roughly in order of yield: evaluator disagreement, where you run two or more judges with different prompts or models and route every disagreement to a human; evaluator uncertainty, where the judge's score sits near its decision boundary or its self-consistency across sampled runs is low; hard rule violations, which are detectable deterministically and always worth surfacing — a promise made, a verification step skipped, a tool called before authentication; outcome signals such as escalations, customer repetition, negative sentiment, or a callback within 24 hours, which need no judgment at all because the conversation tells you it went badly; and novelty, meaning turns far from anything in the labeled set, where the evaluator is least trustworthy.

The honest framing is that you can't know a turn is questionable, only that it's uncertain or anomalous. What that buys you is throughput: it converts "read a thousand transcripts" into "answer fifty yes-or-no questions," which is precisely the fatigue problem the post's lesson is about. The failure mode to watch is that uncertainty sampling systematically misses confident-but-wrong turns, so a small random audit sample has to run alongside it.

“Every clarification from a human should either adjust the agent or its evaluation guidelines going forward”

Adjust in what way? Prompt?

[Source] Verbatim, and the sentence continues "so we don't ask the same question twice." The mechanism is not specified.

[Inference] The sentence names two destinations, and they take different kinds of edit. Adjusting the agent means changing the config bundle, and the post's own framing says a config is much more than a prompt — so the right edit depends on the failure. A wrong tool call is a schema or tool-description fix, not a prompt fix. A missing empathy line is a prompt fix. An illegal path through the conversation is a state-machine fix. A hard compliance rule shouldn't go in the prompt at all; it belongs in deterministic control flow. Adjusting the evaluation guidelines is the other half and it changes the judge rather than the agent: when a human says "no, that response was actually fine," the agent was right and the evaluator was wrong, so the rubric gets amended.

[Search] That second half is what their evaluator post is about — aligning stakeholders on what a behavior like "Agent Assumes Payment" means in a given context, where "each misalignment creates an opportunity to refine the guideline."

[Inference] The requirement hiding inside "so we don't ask the same question twice" is that each clarification has to be stored as a durable artifact with provenance — a rubric clause or a config line — rather than applied once and forgotten. Otherwise the same ambiguity resurfaces with the next reviewer.

“The key is to have a process that rapidly discovers and fixes failures when they occur.”

What part of their pipeline actually provides rapid feedback?

[Source] The post answers this in the very next sentence: "That's why the simulated testing and the post-launch analytics are so vital." No turnaround times appear anywhere in the post.

[Inference] Discovery and fixing have different bottlenecks and it's worth separating them. Discovery is the genuinely fast part, and it's the post-launch layer that delivers it: Trends and Anomalies flags distribution shifts without anyone asking, the Opera behavior evaluators score every conversation rather than a sample, and AI Analyst answers an ad-hoc question in minutes instead of requiring an analyst to pull and read transcripts.

Fixing is where the clock slows. The loop is edit config, run the simulation suite, compare against the last green run, lock the version, push. Simulation is the part that parallelizes, but the post never says how long a full suite takes, and that number is what actually sets cycle time.

[Inference] What's conspicuously absent is anything between "detected in production" and "fixed in the next version" — no hot-fix path, no feature flag, no way to disable one misbehaving skill while the real fix is built. Config versioning and rollback are the closest thing mentioned, but rollback reverts everything rather than the one broken flow. For a voice agent carrying live traffic, that gap matters more than the length of a test run.

“We learned to prioritize requirements by criticality.”

How does this prioritization actually manifest?

[Inference] It should show up in four concrete places. In the release gate, where the criteria quoted in Q29 are "pass rate, no criticals, tool-accuracy targets" — "no criticals" is a hard blocker while pass rate is a tunable threshold, so criticality manifests as a different kind of gate, not a heavier weight. In where the behavior is enforced, since critical requirements shouldn't live in prompt text at all: you cannot reach the account-change state without passing through verification, and that's a graph property rather than an instruction. In evaluation scope, where their companion post's turn-level checks run on every conversation for critical behaviors while style preferences get sampled conversation-level scoring. And in the agent's own trade-off instructions, which is what the post means by guiding trade-offs — telling it explicitly that if brevity conflicts with a required disclosure, the disclosure wins.

[Inference] The open question the post leaves is who assigns criticality and whether the scale is graded or binary. Binary is usually enough and avoids a long argument about tiers.

In general, how would you design this system to ensure that callers aren’t able to use the generic LLM capabilities and are limited to its customer support role? What guardrails?

[Inference] Layer them, because no single one holds, and note which layers are constraints versus requests.

The strongest guardrail is architectural, and their design already provides it. If the agent is a set of narrow skills gated by a deterministic state machine (Q27), there is no state in the graph whose job is "answer general questions," so an off-role request has nowhere to go. A monolithic prompt-driven agent has no such structural limit.

The second is capability limitation: the agent can only do what its registered functions do. If no function issues a refund above a threshold, no amount of persuasion produces one. This is the layer that cannot be talked around.

The third is a small, fast input classifier ahead of generation, labelling each incoming turn in-scope, out-of-scope, or adversarial and routing the last two to a fixed response rather than to the model. The fourth is output filtering — the concurrent guardrail model call their latency post describes, running in parallel with the main call so it costs no response time, checking for off-topic content, unauthorized commitments, and data leakage. Compliance disclosures and verification prompts should be templated strings rather than generated text, since there is then nothing to manipulate.

[Inference] The framing that matters: an instruction in a prompt is a request, not a constraint. Anything you genuinely cannot allow should be enforced by what the system is able to do.

How would you improve this system if you were building it today?

[Inference] Five changes, ordered by how much they would move real quality.

First, fix the evaluation loop, because everything else is measured through it. Freeze a holdout set the config is never tuned against, generate fresh scenarios each cycle, and run several seeds per scenario with a significance test rather than eyeballing red versus green. Without this the pass rate is a vanity metric (Q28).

Second, decompose the simulated visitor. Store persona, world state, goal, and expected outcome as four independent artifacts and sample the cross-product instead of hand-building bundled objects (Q17). This is what makes coverage measurable — you can then report which state-machine transitions the scenario pool has never exercised.

Third, build the human-review sampler the post assumes but never describes (Q33): disagreement between two judges, plus uncertainty, plus a small random audit sample to catch the confident-but-wrong turns that uncertainty sampling misses.

Fourth, split automation readiness into two scores (Q10) — build feasibility and automation risk — and let the risk score act as a veto rather than averaging into a single number.

Fifth, add a surgical rollback path. Being able to disable one skill behind a flag, rather than reverting an entire config version, is what turns "rapidly fixes failures" (Q35) from an aspiration into an operational property.

[Inference] The one thing I would keep unchanged is the decentralized state-machine architecture. It is the decision that makes every other part of the system testable.

“compliant, trusted AI agent”

What are examples of standard compliance requirements for agents?

[INFERENCE] The contact center requirements that actually get encoded as evaluator criteria, grouped by the regime that imposes them:

Debt collection. The FDCPA mini-Miranda ("this is an attempt to collect a debt"), restrictions on contact hours, and no disclosure of the debt to third parties. Regulation F adds call-frequency limits.

Payment handling. PCI DSS forbids storing or logging full card numbers and security codes, which in practice means pausing recording and transcription during card capture.

Health. HIPAA's minimum-necessary rule, plus identity verification before any protected health information is discussed.

Contact and recording. TCPA consent for automated outbound calls, and two-party recording consent in states that require it.

AI-specific. Disclosure that the customer is speaking to a bot, required under California's bot disclosure law and the EU AI Act's transparency provisions, and a working path to reach a human.

The first four predate agents. Only the last is new.

“Enterprises that equate high evaluation scores on OOTB metrics”

What are standard OOTB evaluator metrics?

[INFERENCE] What ships in the common frameworks — RAGAS, DeepEval, Arize Phoenix, LangSmith, and OpenAI Evals — sorted by what each actually measures:

Grounding. Faithfulness, or whether the answer is supported by retrieved context, plus context precision and recall. These are the RAGAS core and the most mature of the group.

Task correctness. Answer similarity against a reference, and task completion.

Safety. Toxicity, PII leakage, bias, and jailbreak or prompt-injection detection.

Conversational quality. Coherence, helpfulness, empathy, tone, verbosity. This is the subjective cluster and the one the post's argument targets.

Agent-specific. Tool-call correctness and trajectory adherence — the newest category and the least standardized.

The post's critique lands hardest on the fourth group, where the construct itself is contested. It applies much less to grounding and tool-call checks, which have something closer to a ground truth.

“Consider the behavior, “Agent Assumes Payment.””

What are other examples of “behavior[s]”?

[INFERENCE] A behavior here is a named, observable conversational act specific enough that a yes-or-no answer is possible. That constraint follows from the post's second principle, and it's what separates a behavior from a quality score. Examples by where they occur:

Opening. Agent verifies identity before discussing the account. Agent states the company and purpose of the call.

Diagnosis. Agent asks a clarifying question before proposing a solution. Agent acknowledges a stated frustration before moving on.

Resolution. Agent commits to a specific next step and timeline. Agent quotes policy without checking the account. Agent over-promises a resolution date.

Compliance. Agent reads the required disclosure before collecting payment details.

Closing. Agent confirms the issue is resolved. Agent offers an escalation path when it isn't.

These read like contact center QA scorecard items, which is almost certainly where the taxonomy comes from — Cresta's origin is human agent quality management.

“ask different stakeholders to label each as a positive or negative example of the behavior”

Are they labeling whether the agent exhibited the behavior or whether it’s good or bad that it did?

[SOURCE] The post is genuinely ambiguous, and you can watch it slide between the two readings. "Positive or negative example of the behavior" is classification language — did this instance instantiate the label. But a few sentences later: "whether it's 'good' or 'bad' depends entirely on the context." That's valence, a different question.

[INFERENCE] The coherent reading is that the labeling task is detection, and the collections-versus-sales contrast is diagnostic of a problem rather than an illustration of the method. In both scenarios the agent performs the same act. Stakeholders resist labeling the sales utterance as "Agent Assumes Payment" because it doesn't feel wrong — their valence judgment is contaminating their detection judgment.

That contamination is worth naming, because it means the disagreement the post attributes to unclear guidelines is partly a task-design flaw. The fix is to split the label into two fields: did the behavior occur, and is it desirable in this context. The first is a classifier; the second is policy. Collapsing them guarantees the inter-annotator disagreement the post then sets out to fix.

“Even among business experts familiar with the space, judgments often differ.”

How to handle disagreements?

[INFERENCE] Four steps, in order:

Measure before resolving. Compute inter-annotator agreement — Cohen's kappa for two annotators, Fleiss' kappa or Krippendorff's alpha for more. The post mentions kappa later for its own evaluators. The number tells you whether the guideline is ambiguous or a particular annotator is an outlier, which have different fixes.

Adjudicate in discussion, not by majority vote. Majority vote produces a label while hiding the ambiguity that caused the split, so the guideline never improves.

Write the resolution down as a rule plus a worked example. The examples transfer better than the abstract rule, and they become the judge's few-shot set later.

Re-annotate a fresh sample to confirm agreement actually rose.

Accept a floor. Some disagreement is irreducible, and there you decide by policy — default to the conservative reading on compliance items — rather than pretending consensus exists.

How to know which/how many examples need multiple reviewers vs single expert?

[SOURCE] Not directly answered, but the post supplies the governing distinction: multiple-annotator alignment is "a key requirement" for models "designed to work across use cases and industries," while aligning to a single expert produces a preference-aligned model that is "easier to define" but has "a more specific scope."

[INFERENCE] That settles the model-level question — cross-customer evaluators need multiple annotators, single-organization evaluators don't. The volume question is separate and the post is silent on it.

You don't multiply-label everything. The normal approach is to double- or triple-label a random subset, often ten to twenty percent, purely to estimate agreement, and single-label the remainder once agreement is acceptable. The subset is a measurement instrument, not the training data.

Beyond that random sample, route specific items to multiple reviewers: those near the decision boundary, those where the first annotator flagged low confidence, and anything in a compliance-critical category where a wrong label is expensive. The logic is stratified sampling — spend extra review where variance is high or the cost of error is high, and accept one pass everywhere else.

I’m not fully convinced that going from a small number of numerical scores to a large number of booleans would solve the inconsistency problem. It feels like we’d have the same amount of room for variation in either case unless we narrow the criteria.

[INFERENCE] Your objection is largely right, and the phrase the post quotes gives it away: the work is being done by precise, not by boolean. Decomposing a ten-point quality score into twenty vague booleans — "was the agent professional" — multiplies the ambiguity rather than removing it. Narrowing each criterion to a concrete checkable fact is the actual mechanism, and that's a change in the construct, not in the response format.

Binary does contribute something real and separable, though. A ten-point scale requires agreement on nine decision boundaries, which the post notes ("one for each decision boundary"). Even with a perfectly clear construct, annotators who agree an example is "good" will still split between seven and eight. Binarizing eliminates that class of noise entirely.

So there are two effects and the post conflates them. Narrowing the criteria is the large one; removing boundary ambiguity is the smaller but genuine one. Neither works without the other.

For Precise Boolean rubrics, what happens if certain criteria aren’t applicable? Like if someone asks an agent for advice, I don’t think grading its factuality would make sense.

[SOURCE] Not addressed. The post presents rubrics as uniformly applied and never mentions inapplicable criteria.

[INFERENCE] You need a third value, and conflating "not applicable" with "no" is a real and common defect. It depresses scores on conversations where the criterion never arose, which makes your aggregate metric a function of conversation mix rather than agent quality — two agents of identical quality score differently because they handled different intents.

Two ways to handle it, which differ in where the logic lives:

Gating. An applicability predicate decides whether the criterion is scored at all. "Was a factual claim made?" gates the factuality check. This is cleaner, because applicability is usually a much easier boolean than the criterion itself.

Three-valued labels. Score yes, no, or N/A, and aggregate as passed over applicable rather than passed over total.

On your advice example specifically, the sharper move is usually to narrow rather than exclude. Advice still rests on checkable premises — "your plan covers this procedure" — so scope factuality to verifiable claims instead of marking the whole conversation inapplicable.

“The only sustainable way to achieve scalable, fully-aligned LLM judges is through continuous human calibration.”

What is the continuous aspect of this?

[INFERENCE] Continuous because three things drift underneath a judge that was calibrated once:

The judge model changes. A provider version upgrade shifts judge behavior, and your agreement numbers were measured against the previous version.

The agent changes. As it improves, the conversations reaching the judge shift toward rarer and harder cases — precisely the region where it was never calibrated.

The standard changes. New products, regulations, and policies redefine what counts as good.

Operationally that means a standing sample of judge outputs re-labeled by humans on a fixed cadence, agreement tracked as a monitored metric with alerting when it falls, mandatory re-validation before any judge model change ships, and newly discovered edge cases folded back into the guideline.

The framing worth carrying: inter-annotator agreement is a production metric, not a one-time acceptance test.

“By injecting human preferences”

What does this mean?

[INFERENCE] Given everything else the post says, this almost certainly means prompt-level calibration: human labels and the clarified written criteria they produce go into the judge's rubric and its few-shot examples. Cresta emphasizes that guidelines are customer-visible and customer-adjustable, which only makes sense if the guideline is text in a prompt rather than weights in a model.

Worth flagging the terminology, because it's borrowed. "Human preferences" in machine learning normally means pairwise comparison data used for RLHF or DPO — you collect A-versus-B judgments, fit a reward model, and optimize against it. Nothing in the post mentions training, reward models, or preference pairs.

The two have very different cost and reproducibility profiles, so the loose usage matters. Rubric calibration is cheap, inspectable, and editable. Preference optimization is none of those. If this comes up with them, it's worth asking which they actually mean.

“Cresta’s guidelines are exposed to customers, so you can understand exactly what each LLM judge is doing, and even adjust the criteria to better fit your organizations’ standards”

In what ways could it make sense to adjust the criteria?

[INFERENCE] Adjustments that legitimately make sense, by what's being changed:

Valence. Keep the detection rule, flip whether the behavior is desirable. This is the collections-versus-sales case exactly.

Strictness. Where the threshold sits — how much hedging still counts as assuming payment.

Applicability. Which conversation types the criterion gates on.

Required elements. Adding a jurisdiction-specific disclosure to a compliant-closing criterion.

Vocabulary. Mapping criteria onto your existing QA scorecard names so results reconcile with the quality program you already run.

One risk the post doesn't raise: adjustable criteria break comparability. Once two customers edit the same evaluator differently, Cresta's "proven to generalize" claim no longer covers either edited version, and cross-customer benchmarks stop meaning anything. Any adjustment should trigger re-validation against human labels, or you've silently decalibrated a judge whose credibility rested on the original guideline.

“OOTB evaluators make big promises about objectivity and scale, but in practice, they deliver metrics detached from meaning.”

This seems plausible but what would be a rigorous quantitative way to measure this?

[INFERENCE] Three tests, distinguished by what each compares against:

Agreement with domain experts. Run the evaluator and your experts on the same conversations and compute kappa or Krippendorff's alpha. Benchmark against human-to-human agreement on that same set, not against one hundred percent — the human ceiling is the only fair target.

Predictive validity. Does the evaluator's score correlate with outcomes that already matter — repeat contact rate, escalation rate, CSAT, compliance audit failures? This is the most direct operationalization of "detached from meaning," and the strongest of the three, because it needs no labels and no one's opinion.

Domain sensitivity. Take matched conversation pairs where experts judge the same behavior differently by context — the collections and sales examples. If the evaluator assigns both the same score, it is demonstrably blind to the thing that determines correctness.

Run all three. The second is where a generic evaluator most often fails without anyone noticing.

“where our iteration velocity started to plummet”

What aspect of their testing process was especially slow and would cause this?

[SOURCE] Partly answered. The anti-pattern section names it: they "over-indexed on brittle, turn-specific assertions and fine-tuning before we had enough test volume to see impact × frequency." The trigger was launching complex billing and survey use cases.

[INFERENCE] The mechanism the post leaves implicit is that the expensive step is human, not compute. A turn-specific assertion hard-codes what the agent should say or do at a particular moment, so every prompt or flow change invalidates a batch of them and someone has to rewrite the expectations by hand. Maintenance cost scales with turns per flow times tests per turn, and billing and survey flows are long — which is exactly why those launches were the breaking point.

Two further costs compound it. Brittle assertions produce false failures on valid rewordings, and each one consumes triage time before you learn nothing was wrong. And because the assertions were written before they knew which behaviors were frequent or consequential, much of that maintenance was spent on behaviors that didn't matter.

So the slow part was rewriting and triaging tests between iterations, not running them.

“able to run 15x more tests with a far greater degree of diversity, leading to 35% faster release cycles”

Why does 15x more tests correspond to only a 35% faster release cycle?

[INFERENCE] There's no reason to expect proportionality, because the two numbers measure different things. 15x is test volume — an input. 35% is cycle time — an output with many other components.

The ceiling is structural. If testing occupies half your release cycle, eliminating testing time entirely caps you at 50% faster. A 35% improvement is consistent with testing having been a large but not dominant share of cycle time. Design, implementation, review, and customer acceptance don't speed up because you ran more tests.

More importantly, the 15x isn't what produced the speedup. Running more tests is, on its own, slower. The speedup came from removing the manual maintenance bottleneck in Q1; the extra volume is what that removal made affordable, and it's what bought the quality gain instead.

Worth flagging: this is a vendor figure with no stated baseline or comparison method. Treat the direction as the claim and the magnitudes as unverified.

“improving accuracy by 20%”

What metric is this?

[SOURCE] Never defined. No metric name, baseline, denominator, or measurement window appears anywhere in the post. This is its least substantiated claim.

[INFERENCE] In this domain "accuracy" could plausibly mean any of several things, and they aren't interchangeable: goal or task completion rate (did the customer's issue get resolved), containment rate (handled without human escalation), per-turn correctness against a golden response, or judge-scored flow adherence. A 20% move means something quite different in each.

The ambiguity that matters most is absolute versus relative. Twenty percentage points (70% → 90%) and a 20% relative gain (70% → 84%) are very different results, and the phrasing supports either.

There's also no stated denominator. Accuracy measured over which conversations — all traffic, the complex billing and survey flows that triggered the rework, or the test suite itself? Improvement measured on your own expanded test suite is partly circular, since you changed what you were measuring at the same time.

Without those, the number isn't interpretable. It's worth asking about directly if this comes up in conversation with them.

“Because this can be automated”

Can’t everything they do be automated? How is dynamic testing unique in this regard?

[INFERENCE] Your challenge is correct as stated — static tests also run automatically once they exist. The post is being loose, and the real asymmetry is in a different place: test case generation, not test execution.

Splitting the pipeline cleanly:

Generation. Dynamic testing produces new conversations from a goal or persona specification, so coverage scales with compute. Static testing requires a human to notice a case, capture it, and define expected behavior, so coverage scales with headcount. This is the only stage where the two genuinely differ.

Execution. Both are fully automated. No distinction.

Grading. Both use the same evaluators — deterministic checks and LLM judges. No distinction.

Calibration and review. Human, for both.

So the defensible version of the claim is that dynamic testing is the only method whose coverage grows without proportional human effort, which is why it can serve as the volume driver. The post compressed that into "can be automated," which overstates the contrast.

“from early design exploration through post-launch monitoring though post launch”

What’s the difference between post-launch monitoring and post-launch?

[INFERENCE] So the substantive claim is about cadence, not about two concepts: run dynamic testing across the entire lifecycle, but expect its intensity to drop after launch and spike again around major updates.

The distinction actually worth drawing — which the post does make, later, in its "Bonus Points" section — is between offline testing and real-world monitoring. Offline testing runs simulated or replayed conversations against known expectations. Real-world monitoring evaluates live production conversations where there is no ground truth. The post says the better your production monitoring gets, the less you need high-volume offline testing.

That's the meaningful post-launch split, and it's a different axis from the one the garbled sentence appears to be drawing.

“Running a large number of simulated customer conversations”

How to determine right number?

[INFERENCE] It's a statistical power question, not a fixed number, and it decomposes into two multiplied quantities: runs per scenario, and number of scenarios.

Runs per scenario. You're estimating a pass rate, so precision scales as one over the square root of the run count. Distinguishing a 90% pass rate from a 95% one with confidence takes on the order of several hundred runs per scenario; confirming a flow is roughly working takes tens. Set this from the smallest difference you actually need to detect. Combined with the prior post's pass^k framing, you also need enough reruns to measure reliability rather than capability.

Number of scenarios. Driven by how many distinct intents and flows you support, weighted by the impact × frequency criterion the post uses elsewhere.

A practical stopping rule beats a formula here: keep adding runs until new runs stop surfacing new failure modes and stop moving your pass-rate estimates. That saturation point is observable, and it's where marginal cost exceeds marginal information.

“replay them against updated versions of the agent to confirm nothing regresses”

Is there any reason to expect regressions? What circumstances make it more likely?

[INFERENCE] Yes, and more than in conventional software, because LLM agents lack module boundaries. A prompt is one global object — editing an instruction for the refund flow changes behavior on the billing flow too, since both are conditioned on the whole prompt. There is no compiler or type system to contain the blast radius, and no way to know what you affected except by testing it.

Other sources, roughly in order of how much they matter: model version upgrades, which change behavior everywhere at once; context pressure, where adding instructions degrades adherence to existing ones; tool or schema changes; and retrieval index updates that change what the agent sees.

Circumstances raising the likelihood: large prompt edits, edits to shared or global instructions rather than flow-specific ones, a model swap, and any flow that was only marginally passing before — those flip on small perturbations.

One caution: a single failed replay may be resampling noise rather than a real regression. That's what pass^k is for.

“Use it selectively early on, only for issues caught during dynamic testing or key “golden” flows.”

Why?

[SOURCE] The post supplies the reasoning across two sections. Turn-level assertions are "brittle when used broadly," and the anti-pattern was building them "before we had enough test volume to see impact × frequency."

[INFERENCE] Two distinct reasons, and it's worth separating them.

Shelf life. Early on the agent's design is still moving. A snapshot freezes behavior at a moment in context, so snapshots written during active design encode behavior that's about to be deliberately redesigned. You pay maintenance cost on tests with short useful lives, and worse, they fail for the right reason — you changed something on purpose — which trains the team to ignore failures.

Selection. You don't yet know which behaviors are worth locking down. "Impact × frequency" is the selection criterion, and neither term is observable until you have volume. Dynamic testing is the discovery mechanism that produces that information.

So the sequencing is deliberate: explore broadly with dynamic testing to find out what matters and what breaks, then spend the expensive per-case curation only on what you've learned is consequential and stable.

“Continue building on test cases as a regression set”

How to build?

[SOURCE] Partly answered. The Bonus Points section supplies the intake source: effective real-world monitoring "allows you to discover additional edge cases and errors to add to your existing static test sets."

[INFERENCE] The pipeline that implies: production monitoring flags a suspected failure, a human confirms it's a genuine defect rather than a judge error, the conversation state at the failing turn is captured, expected behavior is defined, and it enters the suite as a snapshot.

Four things the post omits that determine whether the suite stays useful:

Deduplicate on intake, or one recurring bug fills the suite with near-identical cases and skews what you're measuring.

Tag by flow and severity, so you can run compliance-critical subsets on every change and the full suite less often.

Prune deliberately. When behavior is intentionally redesigned, the corresponding snapshots must be retired or updated. Skip this and the suite accumulates false failures until people stop trusting it — the failure mode that kills regression suites.

Record why each case exists, since a snapshot with no rationale can't be safely retired later.

“This can also include adherence to an overall flow.”

I’m assuming this is being evaluated by an LLM. If so, couldn’t we just put the desired flow in the agent instructions as well? In which case, I’d assume it’s very unlikely for it to deviate from that trajectory.

[INFERENCE] The premise — instructed therefore compliant — is where this breaks, for four reasons.

Instruction adherence degrades with load. A flow spec competes with every other instruction in the prompt, and adherence falls as instruction count and context length grow.

Customers force deviation. Side quests and mid-flow topic changes mean the ideal trajectory often can't be walked straight, and the spec underspecifies how to leave and re-enter it.

Instructions conflict. "Be responsive and helpful" pulls against "complete these steps in order," and the resolution isn't deterministic.

Non-determinism. Even a well-followed flow fails some fraction of runs.

There's also a design reason to keep them separate: the instruction states intent, the evaluator measures realized behavior. An agent that can't follow the flow is equally unable to report reliably that it did, so the check has to be external.

“This approach is resilient to prompt changes”

What does this mean?

[INFERENCE] It means the evaluation criterion doesn't depend on the agent's specific wording. A conversation-level check asks whether the refund was issued and the steps happened in the right order. Reword the prompt, and the agent phrases things differently but the answer to that question is unchanged. A turn-level check asserting specific confirmation wording fails the moment the wording changes, even though nothing is actually wrong.

So "resilient" means a low false-failure rate under changes that don't affect correctness — which converts directly into lower test maintenance cost, the bottleneck from Q1.

The trade-off is sensitivity, and the post is right to keep both scopes rather than picking one. A conversation-level check won't catch a missing disclosure at turn four if the outcome still succeeded. Resilience and precision trade against each other here, which is why compliance work stays at turn level despite the maintenance cost.

“takes the right actions at specific steps of an interaction”

How is this different from the static/snapshot testing?

[INFERENCE] So they compose into a two-by-two, and the reason they feel like the same thing is that one cell dominates in practice.

Static plus turn-level is the compliance combination the post recommends — replay a recorded state, assert the disclosure appeared. This pairing is common enough that the two concepts get conflated.

Dynamic plus turn-level is real and useful: run simulated conversations, then check whether the confirmation followed payment capture in each. You get turn-level precision across generated rather than curated inputs.

Static plus conversation-level is also real: replay a recorded conversation and judge whether the whole flow succeeded.

The clean statement: static versus dynamic determines where the customer turns come from. Turn versus conversation determines what you assert on. The first is about inputs, the second about measurement.

“now real world monitoring becomes critical”

What would that consist of?

[INFERENCE] The structural difference the post doesn't name: offline you have ground truth, in production you don't. Everything below follows from that.

Sampling. Judging every conversation with an LLM is cost-prohibitive at production volume. Stratified sampling weighted toward high-risk flows, with full coverage only where checks are cheap and deterministic.

Operational signals that need no judge. Containment rate, escalation-to-human rate, conversation length, repeat-contact rate within 24 hours, and where customers abandon. These are cheap, continuous, and often detect a regression before any judge does.

Drift detection. Alert on movement in those rates rather than on absolute thresholds, since you have no correct answer to compare against.

Feedback loop. Route confirmed failures into the static suite through the Q9 pipeline.

The judge scores tell you about quality; the operational signals tell you fastest that something broke.

“Large Language Models (LLMs) introduce non-determinism”

What are the different sources of non-determinism in LLMs?

Two groups.

Deliberate sampling — temperature, top-p, top-k, seed. This you asked for and can switch off.

Non-determinism that survives greedy decoding. Floating-point addition isn't associative, and GPU kernels reduce in whatever order is fastest. That order depends on batch shape, and inference servers batch concurrent requests together, so your output can depend on who else hit the endpoint at the same moment. Mixture-of-experts routing, prefix caching, and kernel version drift add to it. Once any of this flips a near-tied argmax, the divergence compounds through the rest of the generation.

For agents, the model isn't the only stochastic part. Tool responses, retrieval indices, and account state all drift. A perfectly deterministic model would still give non-reproducible agent runs.

“meaning the same input can generate an

infinite number of responses”

Is this true even with nucleus sampling?

False as stated, and nucleus sampling makes it more false, not less.

The output space is finite in every case: a model with vocabulary V and maximum output length L can emit at most V^L sequences. For a 100,000-token vocabulary over a few thousand tokens that is astronomically large, but it is a number. Nothing about sampling makes it infinite.

Nucleus sampling then shrinks it. Top-p truncates to the smallest set of tokens whose cumulative probability reaches p and renormalizes, assigning exactly zero to everything in the tail. So the set of reachable outputs is strictly smaller than under unrestricted sampling. Top-k does the same more bluntly.

Your instinct points the right way — top-p is a counterexample to the framing.

The post's argument doesn't depend on the word, though. It needs only "too many valid outputs to pre-script," which holds under aggressive top-p. The imprecision is rhetorical rather than load-bearing, but worth catching: genuine infinitude would imply things about coverage that a large finite space does not.

“In testing, this can create the illusion of success. For example, an AI agent might tell a customer their account issue is resolved simply because the customer insists it is, even though the agent never confirmed the fix in the system. In a goal-driven simulation, this may appear as a 'pass' even though the task was never actually completed.”

Why would this appear as a pass? Shouldn't the evaluator also have access to the system to check if the issue was resolved?

[INFERENCE] Your objection is right, and the post concedes it two sections later: its own remedy is deterministic evaluation using "logic checks like verifying successful API calls, accurate data retrieval." That is exactly the system access you're describing.

So this isn't inherent to goal-driven evaluation. It's inherent to transcript-only evaluation, and it arises when any of three things hold: the harness has no real backend and tools return canned success; the evaluator is an LLM reading the conversation rather than querying state; or "goal met" has been operationalized as the agent's own closing assertion, making the agent its own judge.

[INFERENCE] The principle: never let the evaluated system supply the evidence for its own evaluation. Assert on backend side effects, not transcript claims. Where a real backend is impractical, give the mock inspectable state so you can still assert on effects.

The post would read better framing this as a design error it has since fixed.

“For example, if an AI agent fails to help a customer reset an alarm, it can be unclear whether it forgot to ask them to reset their WiFi or failed to prompt them to check the battery. And when several breakdowns occur at once, it becomes difficult to understand the full scope of what went wrong. You know it failed, but not where or why.”

In this example, an effective AI agent would have conducted request-specific tasks. Since the set of request-specific tasks would naturally vary for different requests, how can you construct a comprehensive and granular evaluation suite that works for a diverse set of requests?

[INFERENCE] Separate what generalizes from what doesn't, so authoring cost scales with intents rather than conversations.

Generic evaluators, written once. Hallucination, tone, compliance disclosure, whether the agent verified before acting, whether it looped. These apply regardless of what the customer wanted.

Per-intent checkpoint rubrics. For each intent, define the checkpoints correct handling must hit. The alarm example becomes a checklist: check power, check battery, check WiFi, confirm the reset took. Store these as data on the test case, not as evaluator code, so adding an intent is a data change.

Derive rubrics from production. Cluster historical conversations by intent and extract the action sequences characterizing successful resolutions. This is the only part that scales to a long tail — and where Cresta's combined human-agent and AI-agent corpus would actually pay off.

Granularity then falls out: a failure reports which checkpoint was missed.

“Even with sophisticated modeling, simulated testing environments often fail to fully represent real-world customer behavior. LLMs—and even human testers—don't always act like real customers, making it difficult to ensure that test scenarios mirror the conversations agents face in production.”

What does this have to do with goal-driven evaluation specifically?

[INFERENCE] It doesn't belong there, and flagging it is correct. This limits simulation as a testing method, not goal-driven evaluation as an evaluation philosophy. The post itself makes the distinction a paragraph earlier, writing that "simulation became a core testing method" in order to enable goal-driven evaluation — a dependency, not an identity.

The post's own taxonomy separates testing methods (static replay, snapshot, simulated customer) from evaluation approaches (deterministic checks, LLM judges, human review). Goal-driven evaluation is an evaluation approach; simulator realism is a property of one testing method. Run goal-driven evaluation on replayed real transcripts and the realism problem disappears, because the customer turns are real.

[INFERENCE] The defensible version: goal-driven evaluation is most valuable when exploring counterfactual agent behavior, which needs a live counterparty, which in practice means a simulator — so in its highest-value application it inherits simulator bias. Real constraint. Flattening it into "limitation of goal-driven evaluation" hides that the fix lives in the simulator.

“verify behavior remains stable across agent updates”

What behavior is being looked at here and how is it assessed?

[SOURCE] The post specifies the input (real historical or test conversations, held fixed) and the purpose (stability across updates), but names neither the behaviors observed nor the comparison mechanism.

[INFERENCE] Holding the conversation fixed is what makes this tractable: replay the customer turns as a constant and regenerate the agent's side, so any difference is attributable to the agent change rather than to conversational drift.

Observable behaviors, in descending order of how reliably you can assert on them: tool calls (which tool, what arguments, what order), knowledge retrieval (which article or record), policy actions (disclosure read, identity verified), and finally free-text response content.

Assessment has to be mixed, which the post leaves implicit. Structured behaviors admit exact comparison — tool name, argument values, and article ID either match the prior version or they don't. Free text cannot, because exact match would flag every valid rewording as a regression, the precise failure the post's opening section criticizes. So free text needs an LLM judge or a semantic similarity threshold against the prior response.

“Captures a moment in context (e.g., when

the AI agent gave an incorrect or best-practice response) for regression testing”

Why do they call it regression testing? What value does this add compared to the full evaluation suite?

[INFERENCE] The term comes from software engineering, where a regression is a previously-working behavior breaking again. The implied workflow: a bad response is observed in production, fixed through a prompt or flow change, and that exact moment is frozen as a test so the fix can't silently revert when something unrelated changes.

Value over the full suite comes in three parts.

Cost and latency. A snapshot is one turn with fixed context; a simulated conversation is many turns with a second model in the loop. You can run thousands of snapshots per prompt change. You cannot do that with simulations.

Diagnostic precision. A snapshot failure localizes to a known behavior. A simulation failure says the conversation went wrong somewhere — the exact complaint the post raises about diagnostic value.

Institutional memory. The suite accumulates a record of every mistake the system has made.

The limitation is symmetric: snapshots only cover failures you've already seen. Hence the pairing with simulation.

“Uses an LLM-based customer that follows a goal or instruction (e.g., booking a flight) under varied conditions, exposing real-world unpredictability and helping identify how the agent handles edge cases”

How do you actually get the simulation to explore more diverse customer trajectories?

[INFERENCE] The levers, weakest to strongest:

Sampling parameters. Raising temperature on the user model is the obvious move and the weakest — it produces paraphrases, not new trajectories.

Persona and goal sampling. Sample over personas (terse, confused, hostile, changes their mind mid-conversation), goals, and hidden constraints the agent must surface by asking. This is the first real gain, because trajectory diversity comes from intent structure, not word choice.

Seeding from production. Draw opening turns, grievance types, and account states from real transcripts, so you test against the distribution you'll actually see.

Environment perturbation. Inject tool failures, timeouts, missing records, stale data, ambiguous account states. Often the highest-yield source, because it forces recovery paths no user-side variation reaches.

Adversarial or curriculum search. Reward the simulator for finding failures and resample around them. The only approach that systematically finds what you didn't think to look for.

The last two account for most of the real coverage.

“Uses logic checks like verifying successful API calls”

How do you verify successful API calls?

[INFERENCE] This collapses three different assertions, and they catch different failures.

Call level — was the right call made? Assert on the structured tool-call record the agent framework emits: tool name, argument values, ordering, and absence of calls that shouldn't have happened. Assert against the framework's record, not the prose transcript, because an agent can narrate a call it never made.

Response level — did it succeed? Status code, absence of an error payload, and schema validation of the body. Schema validation matters more than it sounds: a 200 with a malformed or empty body is a common silent failure.

Effect level — did the world change? Query the sandbox backend afterward and assert on state: the ticket exists with the right fields, the address is updated. This is the layer that catches the Q3 sycophancy case, because it trusts neither the agent's claim nor the API's response.

Most teams implement the first two. The third needs a sandboxed backend with inspectable, resettable state — real engineering investment, not a framework feature.

“Expert-aligned LLM judge evaluation: Employs calibrated LLM evaluators to assess nuanced behaviors, such as flow adherence, response relevance, or factuality”

Calibrated how exactly?

[INFERENCE] What calibration means in practice:

Build the reference set. Multiple experts independently score the same conversations, then disagreements resolve into a consensus label. Multiplicity matters — one annotator gives you no way to know what agreement is achievable.

Measure agreement. Cohen's kappa or Krippendorff's alpha for categorical judgments; correlation or mean absolute error for scalars. Raw accuracy misleads on skewed label distributions, which compliance checks usually have.

Iterate. Adjust rubric wording, few-shot anchor examples, and thresholds until agreement clears the bar. Anchors do most of the work.

Re-validate on change. Judge behavior shifts with the underlying model version, so this is a standing obligation.

“Manual human review: Involves experts validating edge cases, refining evaluator calibration, and ensuring that LLM-based assessments align with enterprise quality standards”

What is an edge case in this context?

[INFERENCE] Two meanings are in circulation and the post blurs them.

Edge case as rare input — unusual account states, rare intents, unusual customer behavior. The conventional software meaning.

Edge case as contested judgment — ambiguous intent, conflicting policies, conversations where the agent was technically correct but the experience was poor, or cases where an LLM judge and a deterministic check disagree.

The second is what the quoted sentence needs, because it concerns where to spend expert human attention. The operationally useful definition: a conversation where your automated evaluators are unreliable — low judge confidence, evaluator disagreement, or a score near a decision threshold. Those are the cases where human review changes an outcome. Rare-but-unambiguous inputs don't need a human; the evaluator handles them.

This definition also makes human review schedulable rather than open-ended. You route the low-confidence tail to experts, and the volume becomes something you can budget.

How do you determine whether it’s “align[ed] with enterprise quality standards”? What metric do you measure and what threshold do you need to meet?

[INFERENCE] Metric. The same machinery as Q10: agreement between the automated evaluator and expert consensus on a labeled sample. But choose it per evaluator rather than globally, because error costs are asymmetric and differ by type. For a compliance check, recall dominates — a missed disclosure violation is regulatory exposure, a false alarm costs minutes of review. For a subjective quality score, correlation or kappa against consensus. For a factuality judge, precision and recall reported separately rather than blended into an F1 that hides which is failing.

[INFERENCE] Threshold. No universal number, and claiming one would be the wrong interview answer. The principled bar is judge-to-human agreement approaching human-to-human agreement on the same data. If two experts agree 80% of the time, demanding 95% from the judge is incoherent — you'd be asking it to outperform its own ground truth. So measure inter-annotator agreement first and treat it as the ceiling.

For high-stakes evaluators, the usual pattern is a recall floor with accepted lower precision, plus routing everything flagged to human review.

“By combining these layers”

How do you combine turn-based and conversation-based layers? Single score (potentially weighed), multiple scores or what?

[INFERENCE] A single weighted score is the intuitive answer and the wrong one, for two reasons. It lets a strong conversation-level outcome mask a turn-level compliance violation, which is backwards for a regulated contact center. And the weights are unjustifiable — there's no principled exchange rate between "read the disclosure" and "resolved the issue."

What works:

Gating. Hard binary gates on compliance and correctness: any violation fails the run outright, regardless of outcome quality. Graded quality scores are computed only among runs that clear the gates. Some failures are not compensable, and this encodes that.

Scorecard, not scalar. Report per-evaluator pass rates as a vector. Any aggregation happens at the reporting layer and never destroys the components.

Structured roll-up. Turn-level results aggregate through explicit operators — any violation, all required steps present, count of loops — chosen per evaluator. This preserves the drill-down path the post's own diagnostic complaint demands.

“must be assessed for both correctness and quality”

Difference between correctness and quality?

[INFERENCE] Correctness is whether the right thing happened, objectively verifiable: the right API called with the right arguments, the right record retrieved, the issue actually resolved in the system, the compliance step performed. Ground truth you can query. Correctness is measured.

Quality is how it happened, and it requires judgment: clarity, efficiency, tone, whether the customer was made to repeat themselves, whether a real person would come away satisfied. No system state to query. Quality is graded, by LLM judges and human review.

[INFERENCE] The post's own two examples fail in opposite directions and are worth keeping as the canonical illustration:

• The flight-booking agent caught in a loop of repeated confirmations is correct but low quality — the booking exists, the experience was poor.
• The sycophantic agent declaring an issue resolved because the customer insisted is superficially high quality but incorrect — it reads smoothly and nothing was fixed.

The second is more dangerous, and it's what a quality-only stack systematically misses. That asymmetry is the argument for keeping both.

“capture both erroneous and "golden" interactions”

Are ‘erroneous’ and ‘golden’ binary or scores?

[INFERENCE] The tension resolves once you see the words doing two different jobs.

As curation labels, binary. Deciding whether a conversation enters the regression suite as known-bad or reference-good is a yes/no promotion decision made by a human. A 0.7 on that axis would mean nothing operationally.

As an evaluation criterion, graded. A Golden Response evaluator compares actual output against the stored reference. Binary exact match would fail on every valid rewording — the precise failure the post opens by criticizing. So it produces a similarity score against a threshold.

So: binary going in, scored evaluating against.

[INFERENCE] A binary curation label also loses information. "Erroneous" flattens a compliance violation and a clunky phrasing into one bucket, which matters if you later want to prioritize the suite or weight failures by cost. A severity field is cheap at curation time and expensive to reconstruct afterward.

“But the majority of people who churn out of SNAP do so from failure to successfully navigate the web of actions required to stay enrolled once eligible”

What is this web of actions?

[SEARCH: 7 CFR 273.12 and 273.14] The recurring obligations are four distinct things, each with its own deadline and its own notice:

1. Change reporting. Under simplified reporting, which most states use, the household must report when gross monthly income crosses the limit for its household size. Under full change reporting, the household must also report unearned income changes over $100, any addition or loss of a household member, residence changes affecting shelter costs, and work hours falling below 20 per week. The deadline is 10 days.
2. The periodic or interim report. A mid-certification form, typically around month six of a twelve-month period. Failing to return it terminates participation.
3. Recertification. A notice of expiration, a new application, an interview at least once every 12 months, and verification documents, generally due at least 10 days before the certification period ends.
4. Work-requirement documentation for able-bodied adults without dependents, which HR 1 extends up to age 65.

[INFERENCE] The word "web" is doing real work here. These are not one process with four steps; they are four independently timed processes whose triggers overlap, and a single life event can put a household into more than one of them at once.

“We conducted a compensated survey with 54 SNAP recipients across 23 states using our voice-based research assistant”

What could the backend of this voice assistant look like?

[INFERENCE] The conventional stack for this is either a pipeline — telephony or in-app WebRTC audio, streaming speech recognition, a language model that holds the interview guide as a loose state machine, then text-to-speech — or a single speech-to-speech model, which cuts the turn latency that makes pipelines feel stilted. The interview guide matters more than the model choice: it has to specify which topics must be covered, when to follow up, and hard guardrails against giving eligibility advice, since a research interview that accidentally advises someone about their case is a liability. After the call you would expect transcription, personally identifiable information redaction, and model-assisted coding of transcripts into themes.

[SEARCH] Several commercial platforms do exactly this kind of AI-moderated interviewing — Listen Labs, Outset, and Strella are the established ones as of 2026. The two metrics the post reports, completion rate and average conversation length, are the metrics such platforms surface natively, though the post does not say whether Propel bought or built.

“Could voice be an effective interface for helping someone report an address change?”

Is the hypothesis that people (a) find it easier to communicate over voice or (b) the address change is more likely to come up over the course of a full conversation as opposed to being proactively volunteered?

[INFERENCE] The evidence in the post supports (b) and points at a third mechanism that I think is the author's actual claim.

Support for (b) is direct: "In several conversations, interviewees reported case changes that had not yet been reported to their state SNAP program." Those disclosures emerged during conversation rather than being proactively volunteered. The strongest single instance is the participant who had given the same address for four years and corrected herself partway through — "I don't live there anymore actually. I'm homeless now." A form would have collected the stale address and recorded it as accurate.

The third mechanism, which the post emphasizes more than either of yours, is translation. Voice lets someone narrate an event in ordinary language and have the system decide which administrative transactions it maps to, instead of requiring the person to pick the correct transaction first. That is about the mapping burden, not about typing ease or recall.

Option (a) is plausible and consistent with the six-minute average completion, but the post offers no evidence for it.

“Although SNAP programs are federally required to deliver notices by mail”

Only by mail or do they do email too?

[SEARCH: FNS guidance on electronic notice options] The claim as written is imprecise. Since November 2017, states may send SNAP notices electronically without a waiver, subject to conditions: the household must consent, the state must provide security protections, and the household can opt back to paper at any time. The permitted combinations are email plus a secure online account, or email plus text. Text alone still requires a waiver, because it gives no delivery confirmation. If an email address becomes inactive, the state must automatically resume paper notices, and paper must remain available on request.

[INFERENCE] The accurate framing is that paper is the default and the mandated fallback, not the only lawful channel. That does not weaken the post's argument so much as relocate it: electronic delivery requires an affirmative opt-in, which is itself one more procedural action that the households most likely to move are least likely to have completed. Brittleness survives the correction.

“Reporting an address change presumes you have an address to report.”

Are there any general mail reception centers that you can put down if you don’t have a specific address? Like a USPS location or something?

[SEARCH] Yes, and a fixed residence is not required for SNAP eligibility. Pennsylvania's SNAP policy manual, which is typical of state manuals, lists as acceptable mailing addresses: a friend or relative's address, a post office box, a social service agency, "any address where the client can get mail without threat of it being lost or stolen," and the county assistance office itself as a last resort.

The USPS option you are describing is General Delivery. Mail addressed to a person's full name, then "GENERAL DELIVERY," then city, state and ZIP, is held at a designated post office for pickup. It is held a maximum of 30 days, pickup requires photo identification matching the name on the mailpiece, and not every post office offers it — availability has to be checked through the USPS location finder.

[INFERENCE] Each of these presumes the person knows the option exists, can reach the pickup point, and holds valid photo ID. Those are the same access constraints that produced the original failure, which is likely why the post frames it as a design problem rather than an information problem.

“we might treat a recent move or other life events as a signal that case continuity is at elevated risk”

Who is ‘we’ here? Propel? If yes, what information do they have access to that would enable this?

[SOURCE] Genuinely ambiguous. The sentence opens "Instead of designing a better address-change process," which reads as a statement to the benefits-design field rather than a Propel product claim. Propel appears elsewhere in the post as the author's employer and as one example of a trusted existing channel, alongside an assistor or a state portal. The post never says what data would drive the risk signal.

[SEARCH] Propel's public product description covers EBT balance on the home screen, transaction tracking, deposit history, and card security features such as locking and blocking out-of-state or online transactions. Propel states it does not store EBT card PINs, complete card numbers, or Social Security numbers, and that it accesses accounts with user permission through state EBT systems. Case status, certification end dates, address of record, and notice history do not appear in that description.

[INFERENCE] So if "we" is Propel, the available signals are indirect: the geography of where transactions occur, a change in deposit amount, or a direct question to the user. That supports a prompt to ask, not a reliable risk score. A state agency, by contrast, already holds the address of record, the certification calendar, returned mail, and notice history, which is where this idea is actually actionable. My reading is that "we" means the field, and the sentence is a design provocation rather than a roadmap.

“we now have an opportunity to use language models that are unusually well suited to translation between highly personalized, unstructured, unique situations (and spoken/written languages) and highly structured administrative systems”

This seems plausible but what is the input layer/UX? Would Propel users have regular AI interviews to get status updates on their life?

[INFERENCE] Recurring scheduled interviews would be a poor design — it would add a new standing obligation, which is the burden the post is trying to remove, and it would collect most of its information when nothing has changed. Three better triggers, which are mutually exclusive and cover the realistic space:

1. User-initiated. The person opens the app with something to do — "I need to report that I moved" — and the conversation starts from their framing rather than a form.
2. Signal-initiated. An external cue starts a short check-in: an approaching recertification date, a deposit that changed amount, returned mail, or a long gap in transactions.
3. Document-initiated. The person photographs or forwards a notice they do not understand, and the assistant explains it and drafts the response.

The post's own requirement that the person confirm and review before submission implies a visible review step, which suggests voice as an input mode inside an app rather than a voice-only product.

“AI could make those systems easier to navigate by absorbing some of that translation work.”

Concrete example?

[INFERENCE, using the reporting rules from Question 1] That one sentence contains three separately reportable items:

1. A household composition change, since the household she is now part of is not the household on file.
2. A shelter cost change, which affects the excess shelter deduction and therefore the benefit amount.
3. An earned income change, which under simplified reporting is reportable only if gross monthly income now exceeds the limit for the household size — and the household size just changed, so the threshold itself moved.

A system absorbing the translation work would recognize all three from the sentence, compute the new income threshold for the new household size, ask only for the hours and wage rate needed to test it, tell her which items actually require reporting and that the deadline is 10 days, name the documents she will need (a lease or rent receipt, recent pay stubs), pre-fill the portal submission, and show her the structured result before anything is sent.

Today, every step of that reasoning is the recipient's unpaid work, and getting it wrong looks identical to not caring.

“It could also make verification, monitoring, and information requests cheaper for institutions, enabling more administrative complexity rather than less.”

Concrete example?

[SEARCH: HR 1 SNAP provisions] HR 1 makes the incentive concrete. Beginning in FY2028, states pay a share of benefit costs scaled to their payment error rate: 5% below a 6% error rate, rising through 15% and 20% to 25% above a 10% error rate. Administrative cost sharing shifts from an even split to 75% state. Work requirements move to ages 17 through 65, the ABAWD exemption age rises from 55 to 65, parents of children aged 7 and above lose their exemption, and exemptions for veterans, people experiencing homelessness, and former foster youth sunset on October 1, 2030.

[INFERENCE] Put those together. A state now has a direct financial reason to drive its error rate down and a much larger population whose work hours must be verified. If an agent can call employers, parse pay stubs, cross-check payroll databases, and generate information requests at near-zero marginal cost, monthly verification becomes affordable where annual verification was the practical ceiling. Every automated request is still a deadline a recipient must meet with documents they may not have. Agency cost falls; the number of procedural failure points rises. One commenter on the post makes the same argument by analogy to automated claims denial in private health insurance — the technology is neutral about which direction it is pointed.

“A person described what happened in ordinary language and the system identified relevant facts”

Does the system require additional context beyond its pretraining/world knowledge? I’m assuming all relevant guidelines are public and digitized, right?

[SEARCH and INFERENCE] Your assumption is mostly right on availability and wrong on sufficiency. Four separate context needs:

1. Federal rules are public and digitized. 7 CFR Part 273 is on eCFR, and FNS guidance documents are on the USDA guidance portal. Most state policy manuals are also on the open web — Pennsylvania's, which I used for Question 5, is an ordinary public web page.
2. But state variation is the whole problem. SNAP is state-administered with wide option-level variation: simplified versus full change reporting, certification length, interview waivers, standard utility allowances, ABAWD waiver areas. A model answering from a pre-trained average will be confidently wrong for a specific state, and HR 1 implementation is changing these rules right now, which makes the pre-training cutoff a live hazard rather than a theoretical one.
3. Some required context is not in any manual. The specific portal's form fields and submission mechanics, what the county office accepts as document intake, and current processing timelines are operational facts, not policy text.
4. Case-specific state is agency data. The certification end date, what is already on file, which notices were sent and when. No amount of document retrieval provides this; it requires an integration with the state system of record.

So: retrieval over versioned federal and state policy with effective dates attached, plus a read integration with the agency. Pre-training alone produces fluent, plausible, state-inaccurate answers, which is precisely the failure mode a high-stakes benefits system cannot absorb.

What are CSAT and resolution rate? How are they measured? Are the numbers available immediately or is there any latency?

From Decagon's own glossary (separate source): CSAT is the share of surveyed customers giving a top rating — their example is 60 of 100 customers rating 4 to 5 out of 5, giving 60% — collected by a short survey delivered "immediately after a meaningful customer touchpoint." Resolution rate is resolved issues divided by total issues times 100, where resolved means the customer needed no further follow-up.

Inferred from background knowledge: the latency question is the important one, and the two metrics behave differently. CSAT arrives within minutes to days, but only from people who respond. Support survey response rates typically sit well below half of contacts, so a per-variant CSAT estimate needs days to weeks of traffic before its confidence interval is narrow enough to act on. Resolution rate is worse, because "no further follow-up" requires waiting out a re-contact window of roughly one to seven days. Until that window closes the measurement is censored, and early readings look systematically better than the settled value. Both are therefore lagging indicators, which is exactly why teams pair them with immediate proxies such as escalation rate, containment, and judge scores computed on live traffic.

“We use models from OpenAI, Anthropic, Gemini, and other providers, alongside our own fine-tuned versions of open-source and commercial models.”

Why so many different models?

Inferred: five reasons, roughly in order of how much they matter.

First, an agent is not one model call. Intent classification, retrieval reranking, guardrail checks, summarization, and final response generation sit at very different points on the cost-latency-quality curve. A small fine-tuned classifier can do intent routing at a small fraction of the cost and latency of a frontier model, and there is no quality reason to use the expensive model there.

Second, enterprise deployments differ in tone, domain vocabulary, and compliance posture, which is what fine-tunes buy you.

Third, their evaluation apparatus makes heterogeneity self-generating. The post says an experiment variant "might involve anything from a new system prompt to an entirely new language model," so if you continuously run model bake-offs per workflow, you end up with a different winner in different places.

Fourth, vendor concentration risk: capacity limits, price changes, deprecations, outages, and data-residency requirements all argue for being able to swap providers.

Fifth, judge independence. Generating with one model family and judging with another avoids the self-preference bias where a model scores its own outputs too highly.

“We rigorously assess every component of the agent, from response generation to retrieval to safety.”

How do you assess safety?

Inferred from standard practice: safety evaluation for a customer support agent breaks into four fairly distinct things.

The first is an adversarial test suite held fixed across releases, covering prompt injection, jailbreaks, requests that would leak another customer's data or PII, out-of-scope advice such as legal, medical, or financial guidance, and brand-unsafe statements. These are scored pass or fail against a written policy, and you measure both directions of refusal: the agent should refuse genuinely unsafe requests and should not refuse benign ones.

The second, and for a support agent the most consequential, is groundedness. The realistic safety failure here is not toxicity but a confident, wrong policy statement, such as telling a customer they qualify for a refund when they do not. You test this by checking that every factual claim in the response is supported by the retrieved context, using either entailment models or an LLM judge given the source material.

The third is action safety. Once the agent can call tools, the question becomes whether it ever took an irreversible action such as issuing a refund or cancelling an order without the required verification.

The fourth is online: guardrail classifiers running inline at inference time, plus continuous sampling of live conversations for review, which is what a Watchtower-style always-on QA product does.

“Every variant must demonstrate consistent performance offline before it is eligible for online testing”

What does consistent mean in this context?

Inferred: two readings are plausible and both are probably intended. The first is consistency across strata, meaning no regression on any sampled slice rather than just a better global average. This reading is supported by the post's emphasis that sampling is stratified across issue types and customer segments; the whole point of stratifying is so that a variant which lifts the aggregate while breaking one customer's billing flow gets caught. The second is consistency across criteria and across repeated runs: no material regression on relevance, correctness, naturalness, or empathy individually, and stable results when the judge is re-run, since LLM judges are stochastic and a single scoring pass has real variance.

In practice this usually gets implemented as a non-inferiority test per slice — the variant must score at least as well as control minus a small tolerance on every gate metric — combined with a genuine win on whatever metric the experiment was designed to move.

“Using an LLM-as-judge system, we evaluate structured triplets consisting of a user query, the context provided to the model, and the model's generated response.”

Only initial user query or intermediate queries as well?

Inferred: almost certainly any turn, not just the first. Two things point that way. The middle element, "the context provided to the model," only has meaning per generation call — it is the assembled prompt for that specific turn, containing retrieved knowledge base passages, conversation history so far, and customer account data. And sampling only opening turns would miss where support conversations actually break down, which is in clarification turns, turns following a tool call, and the decision to escalate.

The limitation worth naming: grading triplets is a static evaluation. Each response is scored conditioned on a history that a human actually produced, so it cannot capture errors that compound across a conversation, where an early turn steers the dialogue somewhere the grader never observes and every later response looks locally reasonable given a history the agent itself would never have reached. The online A/B test is the only part of their described system that catches this, which means failures of that kind are discovered on real customer traffic rather than offline.

“These triplets are drawn from real-world interactions, ensuring that our evaluation process reflects authentic user needs and scenarios”

Sample size and sampling process? How to ensure representativeness/diversity?

Stated in the post: they "sample across key dimensions, such as issue types and customer segments, to ensure balanced testing." No sample size, no selection mechanism, no mention of reweighting.

Inferred: there is a real tension hiding in the word "balanced." Balanced sampling and representative sampling are different objectives. Balancing oversamples rare strata so that each slice has enough data to say something about it, which necessarily distorts the mix away from production. That means the unweighted aggregate score is not an estimate of production quality. To recover a number that describes live performance you have to weight each stratum back by its true traffic share. The post says nothing about doing this, and it is the kind of detail that quietly makes an offline number uninterpretable.

On sizing, the quantity that matters is observations per slice, not the total. As a rough anchor: detecting a 0.1-point shift on a 1-to-5 scale with a standard deviation near 1 needs on the order of 1,600 observations per arm, and resolving a 3-point difference in a pass rate sitting around 90% needs roughly 1,800 per arm. Both numbers drop substantially with a paired design, where the same triplet is scored under both the control and the variant. Since the context is fixed in their setup, pairing is the obvious choice and I would expect them to use it.

“Each response is scored against several key criteria as part of a scalable, continuous, and stratified evaluation process”

Any weighting of criteria? Binary or scores?

Inferred: the criteria are almost certainly scored separately rather than blended, because they are not substitutable. Correctness is a floor and empathy is a preference. If you average them into one number, a warm, fluent, confidently wrong answer can outscore a blunt correct one, which is the single failure an enterprise support vendor cannot ship. The design I would expect, and would argue for, is correctness and relevance as hard gates with pass thresholds, with naturalness and empathy as optimization targets that can only be traded against each other inside the set of responses that already passed.

On the scale: ordinal scores, most likely a 1-to-5 Likert or a three-point pass, borderline, fail scheme, rather than pure binary. Binary throws away information about near-misses, and scales beyond about five points exceed what a judge can apply consistently. Worth knowing that LLM judges are meaningfully more reliable at relative comparisons than absolute ones, so pairwise scoring of variant against control on the same triplet, with position randomized, is the more trustworthy measurement. It does not give you an absolute threshold to gate on, which is probably why an absolute scoring scheme is used here.

If weights do exist, the principled way to set them is to fit them against observed CSAT, so the composite is the thing that actually predicts the business metric rather than a number chosen by intuition.

“We also audit a subset of triplets with human labellers”

What does this mean? What’s being audited?

Inferred: the object under audit is the judge, not the agent. The human label is treated as the reference and the judge's score as a prediction, so the output of the exercise is judge-quality measurement. That means agreement with humans, reported both as raw agreement and as a chance-corrected statistic such as Cohen's or Krippendorff's kappa, and reported per criterion and per slice rather than pooled. Judges are typically strong on relevance and weak on correctness, because correctness requires domain knowledge the judge does not have.

The second function is calibration. If the judge says 92% correct and humans say 85%, you have learned the offset and can either correct the threshold or repair the rubric.

The third is drift detection, and it is the one teams usually discover too late. When the judge model is upgraded, the score scale shifts silently and historical thresholds stop meaning what they meant. A recurring human-audited set is how you notice that the measuring instrument moved rather than the agent.

One thing worth measuring alongside all of this is human-to-human agreement, since it is the ceiling. A judge that agrees with humans as well as humans agree with each other has nothing left to gain.

What model for LLM-as-judge?

Inferred: rather than guess theirs, here are the constraints that determine the choice. It should be a strong model from a family other than the one generating the responses, because judges exhibit self-preference and score their own outputs higher, a well-documented effect in exactly the literature the post cites. It must be pinned to a specific version, because a silent provider-side upgrade moves your metric without the agent changing, and every historical comparison breaks.

Cost is the other constraint. Judging every response with a frontier model is expensive at their volume, so the common pattern is a smaller distilled judge, fine-tuned on frontier-judge plus human labels, doing the bulk scoring, with the frontier model scoring a sample to keep the cheap judge calibrated.

For correctness specifically, a judge without the source material can only assess plausibility, not accuracy. Their triplet format solves this by construction, since the retrieved context is part of what the judge sees.

What prompt/context given to LLM-as-judge?

Inferred: a well-built judge prompt contains six things.

1. It states the role and presents the three triplet elements under clear delimiters, with an instruction to evaluate only the response.
2. It gives one rubric per criterion with each score level anchored by a concrete worked example rather than an adjective, because "3 means partially addresses the question" is not reproducible while a demonstrated example is.
3. It states explicitly that correctness is judged against the provided context, and that a claim unsupported by that context counts as incorrect even when plausible. Without this the judge grades against its own world knowledge and hallucinations pass.
4. It asks for brief reasoning before the score, then structured output such as JSON so results parse reliably.
5. It controls for known biases: randomize ordering in any pairwise comparison, never reveal which variant produced a response, and discourage rewarding length, since judges systematically favor longer answers.
6. It few-shots from the human-audited examples, which has the useful side effect of tying the rubric directly to the human ground truth.

“In parallel, we evaluate responses against a ground truth evaluation set: a curated collection of user queries with ideal responses”

How concretely to evaluate generated responses against ideal responses?

Inferred: the phrase "factuality and intent coverage" describes two different measurements, and they need different machinery.

Intent coverage asks whether the response addressed everything the ideal response addressed. The robust way to measure this is to decompose the reference answer into the discrete facts or steps it contains, then check each one for presence in the generated response. That gives you recall over required elements, paired with a precision check for content the response asserts that the reference does not support.

Factuality is per-claim entailment: extract the claims in the generated response and classify each as supported, contradicted, or unsupported relative to the reference and the source documents.

What not to use is equally important. Exact match, BLEU, and ROUGE punish valid rephrasings and reward parroting, and support answers have many correct phrasings. Raw embedding similarity fails precisely where the stakes are highest, since "you will be refunded" and "you will not be refunded" sit almost on top of each other in embedding space.

One operational issue the post does not raise: ground truth sets decay. When a return window changes from 30 days to 60, the ideal response silently becomes wrong and the benchmark starts penalizing correct answers. The set needs an owner and a refresh cadence tied to policy changes.

“we apply similar rigor across the full spectrum of our AI workflows to ensure each component performs reliably”

How are the other components evaluated?

Inferred, component by component:

• Retrieval is measured with recall and precision at k and ranking metrics such as nDCG against labeled query-to-document relevance, because generation cannot be correct if the right document was never retrieved. The separate and more useful measurement is context sufficiency: whether the retrieved set contains enough to answer at all. That split matters because it tells you whether a wrong answer is a retrieval failure or a generation failure, and those go to different owners.
• Intent classification and routing get per-intent precision and recall with a confusion matrix. The expensive errors are misroutes on high-severity intents such as fraud or cancellations.
• Tool and API calls are scored on recorded traces for whether the right tool was chosen, the arguments were correct, and errors or empty results were handled.
• Multi-step reasoning and policy adherence are tested with scripted scenarios that have required steps in a required order, such as verify identity, then check eligibility, then act.
• Escalation is a classifier decision with its own precision and recall. Both error types cost something: unnecessary handoffs cost money, missed handoffs cost trust.
• End to end, the method that covers what triplets cannot is simulated multi-turn conversations driven by a user-simulator model and scored on task completion.

“We manage rollouts by gradually increasing traffic to the variant group as performance improves”

How concretely to determine when to increase traffic?

Inferred: a defensible ramp has four parts.

There is a fixed ladder of traffic shares, something like 1%, then 5%, 25%, 50%, with a hold at each rung long enough for the lagging metrics to settle, meaning at least one full re-contact window and ideally a full week to average over day-of-week effects.

There are two classes of metric with different decision rules. Guardrails such as escalation rate, safety violations, error rate, and latency are evaluated as non-inferiority tests with automatic rollback when breached, and they move fast enough to read at low traffic. Success metrics such as CSAT and resolution rate are slow and usually only become decisive at higher traffic shares, so they gate the last steps rather than the first.

The statistical point that most teams get wrong: if you look at the data repeatedly and can stop early, ordinary fixed-horizon p-values are invalid and your false positive rate is much higher than you think. The fix is sequential testing — always-valid p-values, mixture sequential probability ratio tests, or group-sequential boundaries — so that continuous monitoring is legitimate rather than decorative.

Finally, promotion should require that no guardrail was breached, not merely that the average improved. An average can absorb a severe regression in one segment, and for a multi-tenant vendor that regression is concentrated on a single customer.

“Customers can also opt out of experimentation, giving them full control over model exposure during evaluation.”

Validity concerns because of selection bias?

Inferred: yes, but it is worth being precise about which kind of validity is threatened, because the two answers are different.

Internal validity survives. Opt-out happens at the customer level and before randomization, and it does not depend on which arm anyone lands in. Randomization within the remaining population is therefore still clean, and the estimate is unbiased for the population that participates.

External validity is what suffers. What you measure is a local effect on opted-in customers, and opt-out is emphatically not random. The customers most likely to opt out are the largest, the most regulated, the most risk-averse, and often the ones with the most complex workflows, which is plausibly the same set where a new model performs worst. So the measured lift is probably optimistic relative to what happens after full rollout, and the bias runs in the direction that flatters the variant.

There is a second problem the post's framing hides. Assignment at the customer level means the effective sample size is the number of customers, not the number of conversations. Conversations within one customer share policies, intent mix, and phrasing, so they are heavily correlated, and conversation-level confidence intervals computed naively will be far too narrow. This needs clustered standard errors or a customer-level analysis.

Mitigations worth asking about: report what share of traffic is opted in, compare the opted-in and opted-out populations on observable characteristics, reweight results toward the full traffic mix, and roll out to opted-out customers in a later, more heavily monitored stage.

“we monitor closely for noise, novelty effects, and performance drift over time”

How are each of these concretely monitored?

Inferred, taking them in order:

Noise is handled by measuring it before you need it. Run A/A tests, splitting control against itself, to learn each metric's natural variance and confirm the randomization is sound, then compute a minimum detectable effect so you do not read differences the design cannot resolve. Variance reduction using pre-period data (the CUPED technique) tightens intervals materially and is nearly free when you have historical metrics per customer. Sequential boundaries keep repeated looks from manufacturing significance.

Novelty effects are detected by plotting the treatment effect against days since exposure rather than pooling the whole period. A novelty or primacy effect appears as an effect that decays or grows over the first days and then flattens, and the rule is to hold the ramp until the curve is flat. Splitting new from returning end users separates the two mechanisms. In a business-to-business agent context it is worth noting that the novelty often sits with the human support agents observing the change rather than with end customers.

Drift is monitored by watching the control arm over time. If control itself moves, something outside the experiment changed: traffic mix, a policy update, an upstream model version, seasonality. Alongside that you monitor input distribution shift in intent mix and query embeddings, and you keep a pinned judge scoring a fixed reference set. If scores on the fixed set move, the measuring instrument drifted rather than the agent, and that distinction is the whole game.

“Insights from every phase feed back into the experiment loop: improving prompts, upgrading models, and refining overall agent behaviors.”

How exactly are these insights used?

Inferred: the version of this loop that actually compounds is error-driven rather than score-driven. A score tells you whether to ship; it never tells you what to change. So the first step is clustering the low-scoring triplets by failure mode — retrieval missed the document, retrieval succeeded but the model answered from priors, tone was wrong, escalation was missed, the tool call was malformed — and sizing each cluster. That converts an uninterpretable three-point score drop into a ranked list of fixes.

Each mode then routes to a different owner. Retrieval misses go to knowledge base content, chunking, or the reranker. Correct-context-but-wrong-answer goes to the prompt or the model. Policy violations go to guardrails. Tone goes to the prompt or a fine-tune.

The mechanism that makes the loop accumulate value is promoting confirmed failures into the ground truth set, so every real defect becomes a permanent regression test and the same bug cannot ship twice. Running the other direction, escalated and low-CSAT conversations from online tests should become the next cycle's offline triplets, which is what keeps the offline set representative as traffic shifts underneath it.

Decagon's own GEPA post is a concrete instance of this loop: prompts optimized against a labeled holdout of roughly 600 examples, reported as a 5 to 6 percent improvement, with the stated discipline being to "treat prompt engineering as software engineering." That is a separate source, not this post.

How would you improve this system if you were building it today?

1. Add conversation-level evaluation. The largest gap is that offline evaluation is entirely turn-level. Triplet grading conditions each response on a history a human produced, so it structurally cannot see errors that compound across a conversation. Add simulated multi-turn evaluation with a user-simulator model and task-success scoring, so those failures get caught before customer traffic rather than during it.
2. Treat the judge as an instrument requiring continuous calibration, not a fixed oracle. Report agreement with humans per criterion and per slice, track it over time, pin judge versions, and re-baseline explicitly whenever the judge changes. An uncalibrated judge score is an unvalidated sensor, and the entire offline gate rests on it.
3. Make correctness a gate rather than a term in an average, for the reason in question 7.
4. Calibrate offline metrics against online outcomes. This is the measurement I would most want that the post does not mention. The open question is how well an offline score delta predicts the CSAT and resolution-rate delta. You can answer it by backtesting every past experiment, comparing the offline predicted effect against the observed online effect. If that correlation is weak, the offline gate is ceremony. The exercise also yields empirically grounded thresholds instead of chosen ones.
5. Reweight the stratified sample back to the production traffic mix, so the aggregate offline number estimates something real.
6. Adopt sequential testing with pre-registered guardrails and automatic rollback, and use clustered inference given tenant-level assignment.
7. Promote cost and latency to first-class evaluation metrics alongside quality. In live support a three-second response that is marginally worse frequently beats a fifteen-second response that is marginally better, and neither appears anywhere in the post.
8. Add offline replay of logged conversations against a new variant as a cheap pre-screen before spending any traffic.
9. Build per-customer dashboards rather than aggregate ones, because a vendor's risk is concentrated per tenant and an aggregate win that hides one customer regressing is a churn risk the average conceals.

“gradient-free optimizer that uses natural language reflection rather than policy gradients to adapt prompts”

What does policy gradient mean?

In RL, the “policy” is the model that chooses outputs. For an LLM, the weights set the probability of every token. A policy gradient method repeats three steps:

1. Sample outputs from the model for many inputs.
2. Score each output with a single number, called the reward.
3. Adjust the weights in the direction that makes high-reward outputs more likely. That direction is the gradient of expected reward with respect to the weights.

GRPO (Group Relative Policy Optimization) is one such method. It samples a group of answers for each input and scores each answer against the group’s average.

GEPA is “gradient-free” because it never changes weights. It edits the prompt text, guided by an LLM’s written diagnosis of what went wrong. A single reward number says how good an output was but not why, which is a big reason policy gradient methods need so many more samples.

“it outperforms reinforcement learning methods like GRPO by up to 20%”

1. By what metric?
2. Is it consistently better across most applications or does it vary?

From the GEPA paper (web, arXiv v2, accepted to ICLR 2026):

(a) The metric is each benchmark’s own score on a 0–100 scale, such as exact match, F1 or pass rate depending on the task. The “20%” is a 19-point absolute gap on HotpotQA with Qwen3 8B (62.33 for GEPA versus 43.33 for GRPO), not a relative gain.

(b) It varies a lot. GEPA beat GRPO on five of six tasks, by 19.0, 13.66, 5.19, 2.73 and 0.69 points, and lost on AIME-2025 math by 6 points (32.00 versus 38.00). The average gain was about 6 points. GRPO was run only on Qwen3 8B, with a fixed 24,000 rollouts.

Inference: “Up to 20%” is the best case. Two of the five wins were under 3 points, which may be within noise. The biggest wins came on multi-hop retrieval tasks (HotpotQA and HoVer), where better instructions about what to search for matter most. The one loss came on competition math, where weight training can build reasoning skill that instructions alone cannot.

“35× fewer model rollouts”

What is a model rollout?

Inference (background knowledge): A rollout is one complete run of the system on one input, followed by scoring the result. For Decagon’s supervisor, one rollout means giving the model one conversation with the current prompt, letting it write its reasoning and label, and comparing that label with the ground truth. The term comes from RL, where it means playing out one full episode with the current policy. Rollouts are the natural unit of cost because each one needs at least one LLM call.

From the GEPA paper (web): Budgets are counted in rollouts. GRPO used a fixed 24,000 per task. GEPA’s total budgets ranged from 1,839 to 7,051, and the paper says GEPA reached its best test performance with 4 to 35 times fewer rollouts.

Inference: GEPA’s total budgets were only about 3 to 13 times smaller than 24,000, so the 35× figure must be counted at the point where GEPA hit its best score, not over its full spend. It is a best-case number.

“We applied GEPA to optimize prompts for a production classification task”

What is the actual task? Input, output and objective?

From the post: The model is a supervisor that “analyzes conversations” and produces structured outputs with reasoning traces. It does binary classification, each prediction must be justified by a chain of reasoning, and the data has ground-truth labels (600+ labeled examples). The post calls it “a last line of defense against hallucinated or inconsistent outputs,” but never names the two classes or the metric.

From Decagon’s guardrails post (web, July 2025): Every message Decagon’s AI agent generates is reviewed by a supervisor model before it reaches the customer. The supervisor checks whether the reply stays on topic, reflects the conversation’s intent and sticks to known facts, and it can revise the reply or escalate.

Inference: Assuming this is the same supervisor, the task most likely looks like this:

• The input is the conversation so far plus the agent’s draft reply, possibly with the knowledge or policy text the agent relied on.
• The output is a reasoning trace followed by a binary verdict, roughly “OK to send” versus “flag.”
• The objective is to maximize agreement with human labels on the holdout set, using a prompt short enough to meet the latency target.

“GEPA operates through four main steps”

Is this cyclical or one-time?

From the post: Cyclical. The section is titled “The core loop,” and the post mentions “Each reflection cycle,” “More iterations,” and a reflection model called “~10-20 times during optimization,” which implies roughly 10 to 20 passes.

From the GEPA paper (web): The loop repeats until the rollout budget is used up. Each pass works like this:

1. Pick a parent prompt from a pool of candidates.
2. Run it on a small batch of training examples and collect feedback.
3. Have the reflection model write an improved prompt.
4. Check whether the new prompt beats its parent on that same batch. Only if it does, score it on the full validation set and add it to the pool.

Parents are sampled from the prompts that achieve the top score on at least one validation example (the “Pareto frontier”), so several different strategies stay alive. At the end, GEPA returns the prompt with the best average validation score.

Inference: The post’s four steps compress a loop that has two gates: a cheap check on the batch, then an expensive check on the full validation set.

“Reflection: A "reflection model" (typically a frontier LLM) analyzes failures and successes, identifying patterns in what works and what doesn't“

All examples fed into a single prompt or multiple?

From the post: Neither all at once nor one at a time. The baseline uses “10 examples per reflection,” and the batch-size ablation tested “how many examples the reflection model should see at once.” So each reflection call sees one batch of 10, and different calls see different batches.

From GEPA’s code, the DSPy docs and the GEPA paper (web): The default reflection prompt packs the current instruction plus, for every example in the batch, its inputs, the model’s response and the feedback into a single call. DSPy’s default batch size is 3, and the paper used 3, so Decagon’s batches were more than three times the standard size. Batches come from the training split, while the validation split is only used for scoring.

Inference: The task model works differently: it is called once per example. Because the reflection model never sees validation examples, validation scores are not contaminated by what the reflector has read.

“Budget: 1.0x multiplier (~150 LLM calls)“

What role does the budget play here? Isn’t the number of calls fixed by the number of examples?

From the post: Budget was one of the seven ablations, with “Diminishing returns beyond 1× baseline.” Calls also scaled with data: about 60 calls for 20 samples and about 150 for 50.

From the GEPA paper (web): The budget caps total rollouts, and the loop keeps cycling until the cap is reached.

Inference: The examples don’t fix the call count, because GEPA is iterative. The examples set the cost of each cycle, and the budget sets how many cycles run. At baseline, the costs are roughly as follows:

• Scoring the seed prompt on the 50 validation examples costs 50 calls, once.
• Each cycle costs 21 calls: 10 to run the current prompt on a batch, 1 reflection call, and 10 to score the new prompt on that batch.
• Each new prompt that wins its batch costs 50 more calls for validation.

So 150 calls buys only about 2 to 4 cycles. That conflicts with the post’s “~10-20” reflection calls, so at least one figure is loose. The “1.0x multiplier” is probably Decagon’s scaling of a budget tied to dataset size, since both 60/20 and 150/50 work out to 3 calls per sample.

“Batch size: 10 examples per reflection”

Does this imply the GEPA process is cyclical/repeated?

From the post: The batch size alone doesn’t prove it, since a one-shot optimizer could also look at a batch. The post confirms repetition elsewhere: “Each reflection cycle sees a batch of examples and proposes refinements,” and the reflection model is called about 10 to 20 times (see #5).

Inference: Yes, and batching is what makes repetition necessary. With 50 training examples and 10 per reflection, no single call sees all the data, so several cycles are needed to learn from all of it. This resembles minibatch gradient descent, which makes many small, noisy updates instead of one update from the full dataset.

Batch size trades off two things. Larger batches let the reflector compare more failures in one call, but they make each reflection prompt longer and more expensive. Smaller batches are cheaper but give a noisier picture of what is going wrong. The post rated this knob “Low impact” and found 10 “sufficient.”

“20–100 is optimal; 500 hurts performance”

Did they test 500 examples with a length-constrained prompt? I wonder if that would fix the overfitting issues.

From the post: No such run is reported. Each experiment “modified exactly one parameter,” and the baseline had no length constraint, so the 500-sample run was unconstrained. It produced prompts 75% longer, performance 2% lower and 10× the compute, which the post attributes to “More iterations with more data.”

Inference: A length cap would likely fix the bloat but only partly fix the performance, for three reasons:

1. The 500-sample run changed two things at once, more data and more cycles, so we can’t tell which one caused the drop.
2. Some bloat is built into GEPA. Its default reflection prompt (web, GEPA code) tells the model to “Identify all niche and domain specific factual information about the task and include it in the instruction.” A cap would make the reflector squeeze niche rules into fewer characters rather than stop chasing them.
3. A 2% drop may be within noise (see #18).

The clean test is a two-by-two design: 50 versus 500 samples, each with and without the 1,500-character cap, holding the number of cycles fixed.

“The optimal range of 20-100 samples provides sufficient diversity”

How are the 20 selected in the first place? Any measures to ensure representativeness?

Inference: It was most likely a random draw, possibly balanced by label. A random 20 carries two distinct risks:

1. It can miss whole groups. If, say, 10% of conversations should be flagged, a random 20 contains 2 flagged cases on average, and about 1 draw in 8 contains none.
2. Even with every group present, which specific examples land in the set can swing the result. The +1% gap between 20 and 50 samples could come from this alone.

Better practice would address three separate goals:

1. For coverage, stratify the draw by label and by known failure type, so every failure mode appears.
2. For learning signal, over-sample cases the seed prompt gets wrong, since GEPA learns mainly from failures.
3. For reliability, repeat the run on several different random subsets and report the spread.

“clear inverted-U relationship”

They say U-shape but the numbers below it seem to suggest it’s monotonically decreasing?

From the post: The bullet list starts at 20 samples, the peak, so it only shows the downslope. The table right below it adds a 10-sample row. Relative to the 50-sample baseline, performance was −5% at 10 samples, +1% at 20, the reference at 50, “Similar” at 100 and −2% at 500. That rises and then falls, which is an inverted U. A regular U would fall and then rise.

Inference: “Clear” is generous, for three reasons:

1. The rising side rests on a single point, the 10-sample run.
2. The gaps among 20, 50 and 100 samples are 1% or less and likely within noise (see #18), so the top is really a plateau.
3. Compute grew with sample size, so the curve mixes a data effect with a cycle-count effect.

A more accurate summary is too little signal below 20, a plateau from 20 to 100, and a mild decline by 500. Also, if the frontier models’ “+5–6% improvement” is measured against the seed prompt in the same units, the −5% at 10 samples means that run barely beat the seed prompt.

What metric is being measured by “Performance”?

From the post: It is never defined. The clues are that the task is binary classification with ground-truth labels, that “All configurations were evaluated on a fixed holdout set,” and that results are reported as percentage changes.

Inference: It is most likely accuracy on the holdout set, meaning the share of conversations whose predicted label matches the human label. It could also be F1, which balances precision and recall. Three further ambiguities matter when reading the tables:

1. The units are unclear. “−2%” could mean 2 percentage points or a 2% relative drop.
2. The reference points differ by table. The sample-size table compares against the 50-sample optimized prompt, the length table against the unconstrained prompt, and the frontier models’ “+5–6% improvement” most likely against the seed prompt.
3. The scope is unclear. The model must justify each label, but nothing says whether the reasoning is scored or only the label.

For a supervisor, a missed hallucination is usually the costly error, so recall at a fixed precision would be more informative than accuracy.

Why is compute cost not linear in # of samples?

From the post: According to the table, it is close to linear. Relative to the 50-sample baseline, 20 samples cost 2.5× less and 500 samples cost 10× more, both exactly proportional, and the bullet counts agree (about 60 versus 150 calls). The deviations are small: 10 samples cost 6× less, where 5× would be proportional, and 100 samples cost 1.8× more, where 2× would be. The post is also inconsistent here, since its bullet list says 100 samples cost “2-4× the compute cost.”

Inference: Near-linear cost suggests the budget was set in proportion to dataset size, about 3 calls per sample (see #7). With a fixed budget, cost would have stayed flat. The small deviations most likely come from rounding in the post’s summary, or from how runs end: GEPA works in whole cycles, so a run stops a little under or over its cap, and that slack is a bigger share of a small budget.

“Substantive evolution”

Qualitatively judged or quantitatively?

From the post: Qualitatively, as far as the post shows. The phrase appears only as a table label for GPT-4.1, GPT-5.2 and the Claude models, next to “No change (65 chars)” for GPT-4o-mini. No measure of how much the frontier models changed the prompt is reported.

Inference: The nearby numbers are indirect. The seed prompt was apparently about 65 characters, since GPT-4o-mini left it unchanged at that length. Unconstrained runs reached 5,000+ characters, and frontier reflection models gained +5–6%. So “substantive” likely means the prompt grew and changed a lot, and performance rose. A quantitative version would do three things for each reflection model:

1. Report how much text changed, such as characters added or edit distance (the number of character edits needed to turn the seed into the final prompt).
2. Report how much the content changed, such as the number of distinct rules added.
3. Report what the change achieved, meaning holdout performance versus the seed prompt.

“the reflection model is only called ~10-20 times during optimization, while the task model (the one executing your prompts) is called hundreds of times”

Sure, but the prompt length for the former is much higher, right?

Inference: Per call, the reflection model has the longer prompt. A task call contains the prompt plus one conversation. A reflection call contains the current prompt plus 10 examples, each with its full conversation, the model’s output and feedback, so its input is several times larger, up to about 10×. That means 15 reflection calls can carry as many input tokens as 75 to 150 task calls.

If the task model ran 150 to 300 times at a similar price per token, the reflection model would be roughly a fifth to a half of input cost, not 5–10%. The claim holds only if the task side is much bigger than the post describes, for example many more task calls, long reasoning outputs, or a pricier task model. So the underlying concern is right: call counts understate the reflection model’s share of cost. The post’s call figures also don’t reconcile with each other (see #7).

“the reflection mechanism naturally accumulates details across iterations”

Any concerns about the order of examples affecting prompt?

From GEPA’s code and paper (web): The default reflection prompt contains only the current instruction and the current batch, with no memory of earlier batches, so the prompt text is the only thing carried forward. Two safeguards exist: a new prompt must beat its parent on the batch before being scored on the full validation set, and the candidate pool keeps several lineages alive.

Inference: Yes, order can matter in two distinct ways:

1. Across cycles, the process is path-dependent. Early batches decide which rules get written first, and later edits tend to add to those rules rather than rewrite them, so a different batch order could produce a different final prompt.
2. Within a batch, LLMs weigh examples unevenly by position, often favoring the first and last, which can shift which failures the reflector addresses.

The safeguards filter out edits that don’t help, but they don’t remove path dependence. Running “19+” experiments across 7 dimensions works out to about one run per setting, so order effects could explain the 1–2% gaps. Several runs with shuffled order would settle it.

“The challenge is GEPA's default implementation doesn't support length constraints during reflection. We built a custom instruction proposer that encodes the constraint directly into the reflection prompt.”

Is this guaranteed to constrain length as intended?

From the post: The limit lives in the wording of the reflection prompt, and no check on the output is mentioned, even though the overview table calls these “hard character limits.” In practice it seemed to work: the 1,500-character limit produced prompts of about 1,000 characters, and the 500-character limit produced about 400.

Inference: No, not by that mechanism alone. An instruction is a request, not a guarantee. LLMs read tokens rather than characters, so they count characters poorly, and a reflector facing many failures may overshoot. The observed undershoot also means the effective limit was tighter than the stated one. A real guarantee needs a check in code after each proposal:

1. Measure the new prompt’s length.
2. If it is over the limit, ask the reflector to shorten it, allowing a few retries.
3. If it is still over, reject the candidate so it never enters the pool.

From the DSPy docs (web): DSPy’s GEPA exposes an instruction_proposer hook for custom proposers, which is the natural place for this check.

“Better generalization by preventing overfitting to training edge cases”

How do you reconcile this with the -0.8% performance impact?

From the post: All configurations were evaluated on a fixed holdout set “to measure generalization,” so the −0.8% is presumably a holdout result. By the post’s own measure, the cap slightly hurt generalization.

Inference: The claim can be rescued in three ways, but the post directly supports none of them:

1. It could mean a smaller gap between training and holdout scores, which is less overfitting even with a slightly lower holdout score. Training scores aren’t reported.
2. It could be a bet that shorter prompts will hold up better on future production traffic, which a fixed holdout can’t show.
3. The −0.8% could be noise. The pool has 600+ examples and one run used 500, so a holdout kept separate from every run likely has only about 100 examples. At 100 examples, one conversation moves accuracy by 1 point, so −0.8% is less than one conversation. At 85% accuracy, the standard error of a single score (its typical swing from which examples happen to be sampled) is about 3.6 points.

The defensible claim is “the same holdout performance with a 4× shorter prompt,” not “better generalization.”

“achieving 4× compression”

Why is this compression valuable?

Compression pays off in three ways:

1. It cuts latency. The supervisor sits in the live reply path, and longer inputs take longer to process, so every extra token adds to the customer’s wait.
2. It cuts cost. Removing about 4,000 characters saves roughly 1,000 input tokens per call (at about 4 characters per token), multiplied by production volume.
3. It eases upkeep. A person can read and audit a 1,000-character prompt, while 5,000 characters of edge-case rules is hard to review and more likely to contain conflicting rules.

How would you improve this system if you were building it today?

Inference (my recommendations, grouped by stage):

Measurement

1. Run every configuration several times on a larger, stratified holdout and report confidence intervals, so 1–2% differences can be trusted.
2. Optimize for the costly error. A missed hallucination usually costs more than a false alarm, so target recall at a fixed precision rather than raw accuracy.

Optimization

3. Enforce the length cap in code (see #17), and replace GEPA’s default instruction to include “all niche and domain specific factual information” with one that asks for general rules.
4. Test interactions, such as 500 samples with a length cap at a fixed budget, rather than changing one factor at a time.

Operation

5. Feed human-reviewed production disagreements back into the labeled pool, and re-optimize on a schedule, promoting a new prompt only if it doesn’t regress on a fixed test set.
6. At high volume, train a smaller classifier to copy the supervisor’s labels for speed and cost, and route uncertain cases to the prompted model.

“RAG is an often-overlooked aspect of agent architecture that can quietly eat into response time.”

How does the query rewriting stage work for voice conversations? If we want to generate embeddings to retrieve relevant context, how do we structure the lookups?

Inference (background knowledge): In voice, the input is a speech-to-text transcript of the caller’s latest turn. Transcripts are messy, with filler words, misheard product names, and references to earlier turns like “what about the other plan?” A small, fast LLM reads the recent conversation plus that transcript and writes one or a few standalone search queries, resolving the references and fixing likely mishearings. Turns like “yes” or “hold on” can skip retrieval.

Lookups usually follow three steps:

1. Indexing: split help articles into chunks, embed each chunk, and attach metadata such as customer, product, and language.
2. Searching: embed each rewritten query with the same model, run nearest-neighbor search filtered by metadata, and run keyword search (BM25) in parallel to catch exact terms like order numbers.
3. Merging: combine the result lists (for example, with reciprocal rank fusion), remove duplicates, and send the top candidates to the reranker.

A voice-specific speedup is starting retrieval on partial transcripts before the caller finishes.

“reranking typically incurs the largest latency cost, as it must process tens to hundreds of candidate documents”

Are they using an LLM for reranking? If yes, why use that instead of a standard cross-encoder or late interaction Models that are more typical in industry?

From the post: The model is never named. Two details are telling. The post says the latency gains came without replacing the underlying model, and the starting point was a list-wise reranker, which outputs a ranked list. The attention-mask diagram also labels the shared prefix “instruction + query” and draws triangular attention inside each block.

Inference (background knowledge): Very likely yes. Producing a ranked list requires a text-generating model, and triangular (causal) masks are how decoder-only LLMs attend; a BERT-style encoder attends in both directions and would show full squares. The setup matches rerankers like Qwen3-Reranker, which put an instruction and query before each document and score it by the probability of answering “yes.” A point-wise LLM reranker is itself a cross-encoder, since it reads the query and document together. The real contrast is a small bidirectional encoder versus a large causal LLM.

Likely reasons for the LLM:

1. Quality: LLM rerankers generally beat BERT-size cross-encoders on public benchmarks.
2. Control: the instruction can carry customer-specific relevance rules.
3. Continuity: the same model already powered their list-wise setup.

Late-interaction models like ColBERT precompute document token embeddings, so they are faster, but they usually rank less accurately than cross-encoders.

“which drives up latency due to the quadratic time complexity of transformer attention”

Even with local attention/other attention optimizations?

Inference (background knowledge): Mostly yes. Attention optimizations fall into two groups, and neither removes the problem for a typical reranker:

1. Exact attention with faster kernels, such as FlashAttention, still computes every token-to-token score. Memory grows only linearly with length and speed improves by a constant factor, but compute stays quadratic.
2. Restricted attention, such as local (sliding-window) or sparse attention, does cut compute to roughly length times window size. But the model has to be trained that way, and list-wise ranking needs long-range comparisons between documents, which a local window undermines.

The post misses two nuances. The quadratic term only dominates once the sequence is longer than about 12 times the model’s hidden size (Kaplan et al., 2020): roughly 12,000 tokens for a hidden size of 1,024, or 49,000 for 4,096. Below that, the feed-forward layers cost more. List-wise reranking also has to generate the ranked list token by token; FIRST (Reddy et al., 2024) reads the ranking from the first output token’s probabilities instead and reports 50% faster inference.

“We recently moved from list-wise to point-wise reranking at Decagon and saw a significant reduction in reranking latency as a result.“

Quality impact?

From the post: No quality results are reported for this switch. The only quality-related claims concern the later packing step: masking keeps point-wise scoring semantics intact and, per the closing section, preserved scoring quality. Neither claim comes with numbers.

Inference (background knowledge): The effect could go either way.

1. Reasons it could hurt: a point-wise scorer never sees the other candidates, so it can’t judge relevance relative to them or notice near-duplicates. In zero-shot prompting (no reranking-specific training), the Setwise study (Zhuang et al., SIGIR 2024) found point-wise methods very efficient but weak on ranking quality.
2. Reasons it could be neutral or help: models trained specifically for point-wise scoring, such as Qwen3-Reranker or RankLLaMA, are strong. Point-wise scoring also avoids list-wise failure modes, like sensitivity to the order in which candidates are listed, and it gives each document an absolute score you can threshold.

The way to find out is to measure ranking quality offline (NDCG@10 and recall@k) on labeled support queries, then confirm downstream effects, such as answer accuracy or resolution rate, in an A/B test.

“The overall reranking step is gated on the slowest individual call, and the overhead of many sequential requests adds up quickly.”

What overhead? And why are the requests sequential as opposed to parallelized?

Inference (background knowledge): The sentence mixes two pictures. A step that waits for its slowest call implies requests running in parallel, while overhead that adds up across sequential requests implies requests running one at a time.

The overhead is most likely the fixed cost each request pays regardless of document content:

1. Transport: serializing the request, the network round trip, and HTTP or gRPC handling.
2. Scheduling: tokenization, waiting in the server’s queue, and batch formation.
3. Compute: fixed costs per forward pass (launching GPU kernels, Python dispatch), plus reprocessing the identical instruction and query for every document.

Why sequential: sending requests in parallel from the client doesn’t mean the GPU runs them in parallel. If a model replica can only fit a few sequences per forward pass, the extra requests wait in a queue and effectively run one after another. Rate limits or concurrency caps on the endpoint have the same effect. Adding replicas helps, but it costs GPUs and still leaves the slowest-call problem described in question 10.

“Here's how it works: we concatenate multiple (query, document) pairs into a single packed sequence, separated by a delimiter token. We then modify the attention mask so that tokens within each pair attend only to each other, while attention to tokens from other pairs is masked out entirely. This produces a block-diagonal attention pattern (see diagram below) — each document is scored in isolation, but we amortize the overhead of a model call across many pairs at once.“

Walk me through a brief illustrative example to make this more concrete. Include relevant math.

From the post: The text describes packing (query, document) pairs, but the diagram shows one shared prefix (instruction plus query) followed by delimiter-separated documents, each attending only to the prefix and its own earlier tokens. The example follows the diagram.

Inference (worked example): Take a prefix (q1, q2) and documents A = (a1, a2) and B = (b1, b2), without delimiters. The mask (1 = may attend, 0 = blocked) is:

q1 q2 a1 a2 b1 b2
q1 [ 1 0 0 0 0 0 ]
q2 [ 1 1 0 0 0 0 ]
a1 [ 1 1 1 0 0 0 ]
a2 [ 1 1 1 1 0 0 ]
b1 [ 1 1 0 0 1 0 ]
b2 [ 1 1 0 0 1 1 ]

The only change from a normal causal mask is the four zeros blocking B from A.

Attention is softmax(QKᵀ/√d + M)·V, where M is 0 for allowed pairs and −∞ for blocked ones. Since e^(−∞) = 0, token b1 spreads its weight only over q1, q2, and itself, and A gets exactly zero. So B’s hidden states match a standalone run on q1 q2 b1 b2, as long as b1 and b2 reuse positions 2 and 3 instead of 4 and 5.

Each score is then read at the document’s last token, typically as p(yes) = e^(z_yes) / (e^(z_yes) + e^(z_no)), where the z values are output logits. The post doesn’t specify the readout.

Cost: two separate calls process 8 tokens; the packed call processes 6, because the prefix runs once.

Why do packed scores exactly equal standalone scores?

The claim is that every document token has the same hidden state, at every layer, as the matching token in a standalone run of [prefix, document]. The argument goes layer by layer (induction):

1. Layer 0. Embeddings depend only on the token and its position. Position encodings are handled separately.
2. Each next layer. Norms and MLPs act on each token separately, so they keep matching states matched. Attention at a document token mixes only the tokens it's allowed to see. The prefix tokens match because the prefix only ever sees itself. The earlier tokens of the same document match by the previous layer's result. Same inputs through the same weights give the same output.
3. Final layer. The final logits match, and so does the yes-or-no score.

What would happen if they retained the batch structure but didn’t do the attention masking in the model call?

From the post: This case isn’t tested. The post justifies packing on the grounds that pairs don’t need to interact to get correct scores, and says masking keeps point-wise scoring intact.

Inference (background knowledge): That framing implies removing the mask would break point-wise scoring. With an ordinary causal mask instead, the effects split into quality and latency.

1. Quality would degrade in two ways. Scores would depend on neighbors: each document’s tokens would attend to every earlier document in its pack, so the same document could score differently depending on what it’s packed with and in what order. At their batch size of 2, the second document in every pack, half of all documents, would be affected. And inputs wouldn’t resemble training data: a point-wise reranker learns from prompts containing one document, so prompts containing several could produce poorly calibrated scores, and later documents would sit farther from the query, adding position bias.
2. Latency would stay about the same or improve slightly, since the work per forward pass is similar and a standard causal mask can use fast fused kernels like FlashAttention.

In short, you’d keep the speed but lose the guarantee that packed scores equal unpacked scores.

Is attention masking generally available through model provider APIs or would you need to host an open-source model yourself for this to work?

Inference (background knowledge): You would need to self-host, or at least run your own inference code. It depends on the type of API:

1. General LLM APIs (OpenAI, Anthropic, Google) accept text, not attention masks or position IDs, so this technique isn’t possible through them.
2. Hosted rerank APIs (for example, Cohere Rerank or Voyage) accept one query plus a list of documents in a single request. The provider can batch internally, which already removes most per-request overhead, but you can’t control the mask or batch size.
3. Self-hosted open-weight models, on your own GPUs or a dedicated deployment that runs your code, are the only option for custom masks. Hugging Face Transformers accepts custom 4D masks together with position IDs, and PyTorch’s FlexAttention can express shared-prefix and per-document masks while skipping fully masked blocks. Serving engines like vLLM support rerankers such as Qwen3-Reranker but don’t expose a per-request mask, so you’d modify the engine or write your own inference loop.

“With this packing approach in place, the remaining challenge is straightforward: find the optimize batch size that minimizes end-to-end latency for your system.“

Quality?

From the post: Batch size is treated purely as a latency knob, with no quality results by batch size. The post argues packing is valid because pairs don’t interact, and it says masking preserved quality, without numbers.

Inference (background knowledge): In theory, with an exact mask and correct position IDs, each document gets the identical score at any batch size, so quality shouldn’t change. In practice, three things can break that:

1. Position IDs: if they aren’t reset for each document, later documents sit at different distances from the query, and scores shift with position in the pack.
2. Numerical drift: different sequence shapes can make GPU kernels add numbers in a different order, which changes logits slightly and can flip near-ties (Thinking Machines, “Defeating Nondeterminism in LLM Inference,” 2025).
3. Truncation: a maximum packed length can cut off long documents in larger packs.

The check they didn’t report is simple: score the same queries unpacked and packed at each batch size, then compare the largest score difference, rank agreement (Kendall’s tau), and NDCG@10.

“where the overall step is gated on the slowest call”

What does this mean?

From the post: The phrase is part of the case against small packs: with too few pairs per batch you send many requests, and the step waits on the slowest one.

Inference (background knowledge): The reranker sends many requests at once, and the pipeline can’t move on until every one returns. So the step takes as long as the slowest request, not the average one. “Gated” means that one slow call holds the gate shut for everything downstream.

For example, if 20 calls return in 40 milliseconds and a 21st takes 250 milliseconds, the step takes 250 milliseconds.

More calls make a slow one more likely. If each call independently has a 1% chance of being slow, the chance that at least one of m calls is slow is 1 − 0.99^m. That is about 10% for 10 calls and 63% for 100 calls. Dean and Barroso describe this effect in “The Tail at Scale” (2013). Packing more documents into each call reduces m, which is the benefit side of the batch-size trade-off.

“Using too many causes the packed sequence to grow long enough that per-call processing time starts to climb.”

What does this mean?

From the post: This is the other side of the trade-off: more pairs per pack means fewer calls, but longer and slower ones.

Inference (background knowledge): A pack of b documents of L tokens each, after a P-token prefix, is about P + b × L tokens long, and longer inputs take longer to process. “Starts to climb” fits how GPUs behave. At short lengths the GPU has idle parallel capacity, so extra tokens are nearly free; once that capacity is used up, time grows with length, and attention grows with the square of length.

Custom masks make this worse. Unless you use a sparsity-aware kernel such as FlexAttention, attention computes a score for every token pair in the pack and only then discards the masked ones.

Example with P = 100 and L = 300: two documents make a 700-token pack with 490,000 attention scores, while five documents make a 1,600-token pack with 2,560,000. That’s 2.5 times the documents for 5.2 times the attention work, and only about 15% of those scores are allowed by the mask.

Bigger packs are also more likely to contain one very long document, which then becomes the slowest call.

Is there any good theoretical explanation for why a batch size of 2 yields the best latency results?

Inference (background knowledge): Nothing in theory singles out 2; it’s an empirical minimum. A simple cost model does explain the U-shape if total latency tracks total compute (say, calls queuing on a busy GPU). With N documents of L tokens in packs of b:

Latency ≈ N × (F / b + k × L² × b) + constant

F is each call’s fixed cost (overhead plus the shared prefix), and k × L² × b is attention cost that grows as packs lengthen. The first term falls with b and the second rises, so latency is lowest at b* = √(F / (k × L²)). Longer documents push that optimum down.

Batch size 2 beats 1 and 3 when F is 2 to 6 times k × L², meaning each added document is costly relative to a call’s fixed cost. That fits dense custom masks; a sparsity-aware kernel would shrink k and raise the optimum.

With one measurement per point, 1 versus 2 may be noise.

The post seems to focus primarily on the benefits of this approach. Are there any downsides that they haven’t covered?

Inference (background knowledge): The uncovered downsides fall into four groups:

1. Quality: point-wise scoring can’t see other candidates, so it can’t penalize near-duplicates or reward coverage. Neither change comes with quality metrics, and position-ID handling isn’t described, though getting it wrong silently shifts scores.
2. Efficiency ceiling: an optimum of 2 still means making half as many calls as there are documents. If dense custom masks are the cause, speed is being left on the table, and the optimum has to be re-tuned whenever the model, hardware, or document lengths change.
3. Engineering cost: custom masks require self-hosting and custom inference code, can break during library upgrades, and may bypass serving-engine features like continuous batching and prefix caching.
4. Evidence: without latency values, percentiles, or error bars, the size of the win is unknown. Batch size 5 is slower than the list-wise baseline (per the plot), which goes unexplained, and tail latency (p95 and p99), which matters most for voice, isn’t reported.

How would you improve this system if you were building it today?

Inference (my recommendations): I’d make four changes, roughly in order of payoff:

1. Send fewer documents to the LLM. Add a cheap first-pass scorer, such as a small cross-encoder or a ColBERT-style model, to cut the tens-to-hundreds of candidates to about 20. Also cache scores for repeated query-document pairs, which are common in support.
2. Make each pass cheaper. Compute the instruction-plus-query prefix once and reuse its key-value cache (the attention keys and values already computed for those tokens) for every document, and use a sparsity-aware kernel like FlexAttention so cost tracks each document’s own length. That should push the optimal batch size well above 2.
3. Shape the calls and their timing. Pack by token budget rather than a fixed document count, so no call is much longer than the rest, and start retrieval on partial transcripts while the caller is still speaking.
4. Measure it. Track p50, p95, and p99 reranking latency, and gate every change on packed-versus-unpacked score equivalence plus NDCG@10 on labeled support queries.

“When a company receives a contract from a counterparty, someone in their legal department compares it, clause by clause, against their company’s playbook”

How long are company playbooks typically? Are they only text or can they contain other kinds of data too?

[Post] Playbooks "can be hundreds of pages long." A single rule has five parts: the standard position (what the company wants), acceptable deviations (what it will settle for), unacceptable deviations (dealbreakers), guidance on how to apply the rule, and whether the clause is allowed to be missing entirely. Their test set covers "hundreds of provisions."

[Search] The real range is wide. Practitioner guides suggest a first playbook is about an afternoon's work covering the ten or twenty terms that account for most of the redlines a team actually sends. A mature playbook for a single contract type is more like thirty to eighty terms across a few dozen pages. Harvey's "hundreds of pages" describes a large company maintaining playbooks for many contract types at once.

[Inferred] They aren't pure prose. A real playbook usually contains a ladder of fallback positions — ask for this, settle for that, walk away at this — along with the exact wording to paste in for each fallback, dollar thresholds that determine which position applies, and a table of who has to approve what. Harvey's five-field structure suggests they store playbooks as structured records rather than as documents to be read. That matters later, because it's what lets them hand one rule to one agent.

“Playbook rules are conceptually simple, and include the following elements”

How generic are these rules? Is there a single playbook for all kinds of contracts?

[Post] There isn't one playbook for everything. The test data "pairs contracts with playbooks for various contract types," so a playbook belongs to a type. Rules are written generically and then interpreted per deal, which is what the guidance field exists for: "how to interpret and apply the rule given specific deal context."

[Inferred] The normal setup is one playbook per contract type — NDAs, master services agreements, data processing agreements, order forms, vendor agreements. They have to be separate because the same term means different things in each. A liability cap that's perfectly sensible in an NDA would be unacceptable in a services agreement worth millions.

Some terms travel well across types: governing law, how notices get delivered, whether the contract can be assigned to an acquirer. Teams often keep a shared set of those and layer type-specific rules on top.

The practical point is that a rule as written is deliberately incomplete. "We prefer mutual indemnification" doesn't tell you what to do on a particular deal. What fills in the blank is the deal context — whose paper the contract is on, which party you represent, how far into the negotiation you are. That's exactly the information the old system kept losing between steps, and that the new one broadcasts to every agent.

“Rules interact across a document”

Is it possible for rules to be contradictory? If so, what happens?

[Post] Yes, and the post is direct about it. In the old system, "two rules could propose contradictory edits to the same sentence, and there was no rewriting for those edge cases" — the user sorted it out by hand. In the new system, when two rules try to change the same text, the merge step flags it and hands it to the lead agent, which writes a single edit "that satisfies all conflicting rules."

[Inferred] Two different situations are being treated as one here, and only the easier one is actually solved.

The common case is two rules that both want to change the same sentence but aren't really opposed. One rule wants a notice period extended from 15 to 30 days; another wants notices delivered by email as well as post. Both can be satisfied by one carefully written sentence, and that's what the lead agent is doing.

The rarer case is two rules that genuinely cannot both hold — one requires uncapped indemnity for data breaches, another caps all liability at the contract value. No wording satisfies both. The post never addresses this. It's really a defect in the playbook rather than in the contract, and the right response is to stop and tell a human, which fits Harvey's own point that "some decisions cannot be delegated." But nothing they describe would detect it, so the lead agent would presumably draft something plausible and move on.

“it's a quality bar that out-of-the-box AI models and agents miss”

This feels fixable by having “The best edit is usually the smallest one” included as instructions in the system prompt. Would that not work?

[Inferred] Adding that line helps, and they almost certainly had something like it already. It doesn't fix the problem, for two reasons.

The first is that telling a model to make a small edit doesn't help it work out which edit is the small one. To change only the three words that matter, you have to know which three words create the legal problem. That's a hard reading task, and an instruction to "be minimal" gives the model no extra help with it.

The second is that the old setup made minimal edits nearly impossible to produce. It asked the model to rewrite the whole clause, then compared the new version to the original afterward and called the difference a redline. Models trained as helpful assistants tidy as they write — they smooth grammar, reorder for clarity, standardize phrasing. Every one of those shows up in that comparison as a proposed change, even though none of them was the point. The instruction to stay minimal was fighting the model's default writing behavior, and losing.

The new system asks for something different: not "rewrite this clause" but "change these specific words in this specific place." That's a change to what the model produces, not a change in how politely you ask for it.

“still produce edits in a timely fashion”

What are the realistic latency constraints/expectations here? Contract review feels like it should prioritize quality over latency.

[Post] 2.6 minutes before, 3.8 after, which Harvey calls "a reasonable tradeoff to make as we are optimizing for the quality of output." A human review takes 20 minutes to five hours. They also added streaming so users "begin reviewing outputs within seconds."

[Inferred] You're right that quality should win, and Harvey clearly agrees — they accepted a 47% slowdown to get the quality gain. But latency still matters, for reasons that have nothing to do with impatience.

The one that matters most is the first ten seconds. If a lawyer uploads a contract and sees a blank screen, they assume it's broken, or they go do something else and lose the thread. Streaming fixes that without making the review any faster overall, which is exactly why they built it.

The second is that nobody runs the review once. A reviewer changes a deal instruction, or decides to push for the standard position instead of the fallback, and runs it again. Five of those at four minutes each is twenty minutes, which puts you back in the range of just doing the review yourself. That's why the system saves each agent's work and reuses it rather than starting from scratch.

So the constraint isn't a hard number. It's roughly: something useful on screen within seconds, and the whole thing done in less time than it takes to get a coffee.

“For each batch of rules, one model call asked: Does the contract match the rule's standard position?”

Are multiple rules being reviewed simultaneously or one at a time?

[Post] Multiple at once, in batches. One model call took a group of rules and asked, for all of them, whether the contract matched the standard position. Rules that didn't match fell through to a second call about acceptable deviations, and those that failed again fell through to a third about unacceptable deviations.

[Inferred] Batching was there to save money. The contract is the expensive part of the prompt — you pay to send the same fifty pages every time — so asking about twenty rules in one call is far cheaper per rule than twenty separate calls.

The cost is exactly what Harvey complains about. All twenty rules share one call's worth of thinking. A rule that requires tracing a definition across four sections gets the same shallow treatment as a rule that a single sentence settles. In their words: "a single LLM call with limited budget won't be able to think through and give accurate output to solve complex documents and playbooks."

The new design flips this. Every rule gets its own agent, and they run at the same time instead of being crammed into one call. The savings that batching used to provide come back a different way: because every agent reads the same contract, the model provider charges full price only for the first one and a fraction of that for the rest. Same economics, without making rules compete for attention.

“Once classifications were done, a separate retrieval step then located supporting text in the document”

How would this work?

[Post] The post doesn't say. We only get that after classification, "a separate retrieval step then located supporting text in the document," feeding a final call that wrote the summary.

[Inferred] The job is to find the exact piece of contract that justifies the classification, so the interface can highlight it and the lawyer can check the reasoning. There are three normal ways to do that.

The simplest is to have the model quote the relevant text, then search the document for that quote to find where it lives. Cheap, and it works right up until the model paraphrases slightly or invents the quote, at which point you find nothing.

The second is semantic search: split the contract into chunks, convert both the chunks and the rule into numerical representations that put similar meanings close together, and return the nearest chunks. This survives paraphrasing but tends to give you a rough region rather than the exact sentence.

The third, and most reliable, is to number every clause up front and have the model return clause numbers instead of text. That's what the new system does everywhere — "we gave each component of the document a unique identifier."

Why it was a separate step at all: the classification call probably didn't have the whole contract in front of it, so the location had to be recovered afterward. And doing it afterward is precisely how you get the failure they describe, where a suggestion lands on the wrong part of the document.

“We diffed the revision against the original, and that diff became the tracked change.”

What does tracked change mean here?

[Inferred] A tracked change is Word's redline — text marked as inserted or deleted, attributed to an author, each one individually acceptable or rejectable by whoever reads it next. That's what the lawyer sees and what goes back to the other side.

What's interesting is that the phrase means two different things in the two systems, and that difference is the whole design.

In the old system a tracked change was an output. The model wrote a new version of the clause, code compared it to the original, and whatever fell out of that comparison became the redline. Nobody could tell which part of the change was the substantive fix and which part was the model tidying grammar along the way.

In the new system a tracked change is what the agent creates. An agent doesn't hand back replacement text; it hands back "in clause 8.2, replace these words with those words, because of rule 14." Once edits are objects like that, you can tell which rule caused which change, detect that two agents want to touch the same words, and later learn from which edits the lawyer kept. None of that is possible when the edit is just a comparison you ran afterward.

“The core dataset pairs contracts with playbooks for various contract types across hundreds of provisions.”

Synthetic or human generated?

[Post] Not stated. Only that they "built our own evaluation suite with our in-house legal team," that the dataset "pairs contracts with playbooks for various contract types across hundreds of provisions," and that the scoring rubrics were written with lawyers.

[Inferred] Almost certainly a mix, with the pieces coming from different places.

The contracts are most likely real ones from public sources. Companies file their material contracts with the SEC, and those filings are the standard source for anyone building legal AI evaluations. Customer contracts usually can't be used, because their confidentiality terms don't permit it.

The playbooks are most likely written by Harvey's own lawyers to pair with those contracts. A customer's real playbook is a competitive asset and generally isn't available for this.

The rubrics are explicitly human — "after sourcing the rubrics with lawyers." The grading itself is done by models.

This matters more than it sounds. If the same organization wrote the playbooks and designed the system, those playbooks will be cleanly structured and unambiguous in exactly the ways the system handles well. Real customer playbooks are messier: inconsistent, partly self-contradictory, written by different people over several years. The improvements are real, but they were measured on the friendlier version of the problem, and the post gives you no way to tell how much friendlier.

“Risk classification: Ability to identify and categorize issues by their level of risk.“

Accuracy, precision, recall?

[Post] Not stated. One number, 59% rising to 77%, and a description of the task as "a classification problem which is a straightforward task."

[Inferred] A single percentage on a task with three possible labels is almost certainly plain accuracy — the fraction of provisions where the system's label matched the lawyer's. Three things make that a weak way to report this.

Most clauses in most contracts are fine. If 80% of provisions match the standard position, a system that labels everything "standard" scores 80% and catches nothing. Accuracy hides that entirely.

The two kinds of mistake also cost wildly different amounts. Flagging something that was actually fine wastes a few minutes of a lawyer's time. Missing a dealbreaker can cost real money. Harvey knows this — their own list of old failures names both "over-redline" and "miss flags" — so they almost certainly track the two separately internally. They just didn't publish it.

And the three labels are ordered: standard, acceptable, unacceptable. Calling an unacceptable clause "standard" is far worse than calling it "acceptable," because in the first case nobody looks at it again. Accuracy scores both as equally wrong.

The number I'd want is how often the system catches the unacceptable ones, reported on its own.

“Redline quality: A list of considerations that a revision by a competent lawyer would have, both in substance and style. These rubrics are defined for each example.“

Binary or score? Weighting of different dimensions?

[Inferred] "A list of considerations… defined for each example" strongly suggests a checklist per example, where each item is a yes-or-no question the grader answers, and the score is the fraction of items passed.

This is the standard way to build these, and it exists to avoid a specific problem. If you ask a model to rate an edit from 1 to 10, you get different answers on different runs and no way to explain the difference. If you ask "does this edit change only the words that needed changing? yes or no," you get something stable enough to compare across versions of the system.

Weighting is then probably implicit rather than declared: a dimension gets more weight by having more checklist items about it. So a rubric with four questions about legal correctness and one about placement effectively weights correctness four to one, without anyone writing down a weight.

The design I'd expect but that Harvey doesn't claim is a hard failure condition on legal correctness. An edit that is beautifully minimal, perfectly placed, and legally wrong should score zero, not two-thirds. Whether they do that changes what 87% actually means, and there's no way to tell from the post.

“We then used a committee of three frontier models to score independently and aggregate the votes.“

Which models?

[Post] Not named.

[Search] Harvey deliberately runs on models from three companies — OpenAI, Anthropic, and Google DeepMind — and their published lineup names Claude Opus 4.6 (for "agentic reasoning and deep multi-step legal analysis"), GPT-5.2, Sonnet 4.6 (for lower latency), and Gemini 3 Flash (for speed and throughput). The obvious reading of "three frontier models" is one top-end model from each of those three companies. (Source: harvey.ai/blog/why-harvey-is-multi-model-by-design)

[Inferred] The part that actually matters isn't which three, it's that they come from three different companies rather than three sizes of the same model.

Models from the same family have been trained on similar data with similar preferences, so they tend to make the same mistakes and like the same writing. Three of them voting doesn't give you three opinions, it gives you one opinion repeated. Spreading across companies gets you genuinely different judges.

That's especially important here because the system being graded is itself built on one of those companies' models, and models rate their own family's output more generously. Mixing providers reduces that, though it doesn't remove it — one of the three judges is still likely from the same family as the system under test. More on this in Q16.

“equipped with tools to search for various key parts of a document”

What kinds of tools? How do they work?

[Post] Three kinds are named: searching the document, looking up rules and other information, and editing the document. The thing that makes all of them work is that "we gave each component of the document a unique identifier," which "allows agents to cite and edit a specific element unambiguously."

[Inferred] The likely tool set breaks into four groups.

Reading and navigating: fetch a specific clause by its number, get the table of contents, read a whole section, follow a cross-reference or jump to where a defined term is defined.

Searching: keyword or meaning-based search over the contract that returns clause numbers rather than raw text, so every result is something the agent can point at precisely.

Editing: propose a change to a specific piece of text in a specific clause, delete something, insert something after a given clause. Each one gets recorded as a redline tagged with the rule that caused it.

Reporting: file the memo at the end with the classification, the position chosen, and the reasoning.

The reason the identifiers matter so much is worth spelling out. Without them, the only way a model can tell you where it means is by quoting the text. That breaks in three routine ways: contracts repeat near-identical language in several places, the model paraphrases slightly when quoting, and sometimes it invents a quote outright. A clause number either exists or it doesn't, and it points at exactly one thing.

“but was very high latency”

What's a reasonable guess for the latency difference between the single-agent and the multi-agent systems?

[Inferred] You can reason it out from the structure. In the multi-agent system, all the per-rule work happens at once, so the total time is roughly: set up, wait for the slowest single rule, merge, final check. It doesn't matter much whether there are 20 rules or 60.

A single agent does the same work one rule after another. With 30 to 60 rules and a couple of minutes of work per rule, you land somewhere between half an hour and two hours — call it ten to thirty times slower. That's not "a bit slow," that's back inside the range of a human doing the review by hand, which is what makes it unacceptable rather than merely annoying.

Two things make it worse than the simple multiplication suggests. The single agent's context grows with every rule it handles, so it gets slower and worse as it goes — the post's own point that quality degrades as you add tokens. And it's one long chain, so a single stuck step stalls everything, with no way to retry just the rule that broke.

Any concerns about using LLM-based systems to both solve and evaluate?

[Post] The risk isn't discussed. What they do describe as safeguards: the rubrics were written by lawyers rather than models, the graders score against fixed rubrics rather than giving free-form opinions, and three models vote independently.

[Inferred] There are real concerns, and they enter at different points.

Models rate their own family's output more generously than other models' output. Using three different providers reduces this but doesn't remove it, since the system under test is presumably running on one of those three families.

More worrying is what all frontier models have in common. They've been trained to prefer writing that is fluent, thorough, and complete. That's precisely the instinct that "lightest touch" is fighting. A grader with that instinct will quietly reward an edit that rewrites the whole clause nicely over one that changes three words, which means the redline score can climb while deals don't actually move any faster.

There's also the plain fact that 53% to 87% was measured against the rubric the team was optimizing toward. Any metric you tune against stops being an independent measurement of the thing you care about.

And the post reports nothing about whether the graders are any good — no figure for how often the model graders agree with lawyers, and none for how often the three models agree with each other. Without those, you can't tell how much of that 34-point gain is real.

What I'd want: lawyers who didn't write the rubrics grading a held-out set blind, plus the rate at which real customers accept the redlines in production.

“lead agent in charge of distributing work between parallel agents”

What does the lead agent tell the parallel agents to do?

[Post] It "spawns a team of subagents to review each playbook rule." Each one receives the shared deal context — "which party they represent, whether the contract is on your template or theirs, how strict or permissive this negotiation should be, and any deal-specific instructions" — plus any precedent documents the user attached. The subagent then "reads its rule, reads or searches the contract, decides whether the contract complies with the playbook, chooses which position to pursue (standard or a specific fallback), and then drafts a list of suggested edits," finishing with a short memo.

[Inferred] So what actually gets handed over is roughly four things: the one rule it owns, in its five structured fields; the shared deal context; a pointer to the contract — its own branch and the clause numbering — rather than the contract text pasted in, which is what makes the caching work; and a description of what to hand back.

What's notably absent is any information about what the other rules found. The subagents run at the same time and are blind to each other. That's a deliberate choice, and it's the reason reconciliation exists at all: the lead agent's job is to divide the work up front and merge it at the end, not to coordinate anyone in between.

“lead agent reconciles the results to ensure final output is of good quality”

What does reconciliation look like concretely? How does it work?

[Post] It happens in two stages. The merge "merges branches the same way a version-control system would: non-conflicting edits apply cleanly to the final doc, and collisions of two rules touching the same text are escalated for the lead agent to resolve." Then the lead agent resolves those conflicts "the way a senior lawyer would by drafting edits that satisfies all conflicting rules," and "takes a final pass to validate the quality of the review as a whole. This ensures that every rule gets classified and has corresponding edits when needed."

[Inferred] The important thing is how little of this involves a model.

Most of it is ordinary code. Each agent's edits are recorded against specific clause numbers and specific character ranges within them. To merge, you check whether any two agents touched overlapping text. If they didn't — which is the case for the large majority of rules — you apply everything and you're done. No model call, no cost, no risk.

Only the overlaps go to the model. For each one, the lead agent sees the original text, the two competing edits, and the memos from both agents explaining what they were trying to achieve, and writes one edit that does both jobs. This is what those memos are for, and it's why they need to be short: they're the only channel through which one agent's reasoning reaches the lead agent.

Then the final pass checks coverage — every rule has a verdict, and every flagged rule has edits — and reads the merged document as a whole.

The gap is that conflicts are found by checking whether two edits overlap in the text. That catches collisions but not everything that can go wrong. See Q20.

“running up to dozens in parallel”

How does it determine how many need to be run? Based on the number of rules?

[Post] Yes — one agent per rule, always: "every rule in a playbook fans out a subagent." The ceiling on how many run at once comes from infrastructure rather than from any planning: "we added concurrency limits and exponential backoffs to avoid overwhelming shared model infrastructure."

[Inferred] So there are two different numbers. The number of agents needed equals the number of rules, which could be in the hundreds — their own test set covers "hundreds of provisions." The number running at any given moment is capped, with the rest waiting in a queue. "Dozens" is describing the cap, not the size of the job.

This is a deliberately simple design, and it's worth noticing what it gives up. There's no step that decides some rules aren't worth an agent, and no grouping of related rules. If a playbook has four separate rules about limitation of liability, four agents each read the same section, each build their own understanding of it, and then three of the four collide with each other and have to be cleaned up afterward.

What they get in exchange is that the behavior is completely predictable, and caching makes all that duplicated reading nearly free. That's a reasonable trade, but it's a trade — I'd revisit it, see Q37.

How does the multi-agent architecture deal with concerns about rules interacting across a document? Especially since they say “[sub-agents] often would need to apply edits to the same paragraph or sentence while addressing different rules”

[Post] Three mechanisms, doing three different jobs.

Agents can't corrupt each other's work, because each one edits its own copy of the document. Without that, one agent's changes would silently overwrite another's — the post's exact concern: "previous work done by one agent would get overwritten by another agent."

Overlapping edits get noticed rather than silently resolved, because the merge step surfaces them as conflicts.

And conflicts get resolved by the lead agent writing a single edit that satisfies both rules, followed by a read of the whole document.

Separately, a single rule that touches four sections is handled inside one agent, since the agent can search the contract and follow references. That's the direct fix for the old system's problem that "edits were scoped by clause."

[Inferred] Sorting out what this does and doesn't cover:

It genuinely solves agents overwriting each other, and it solves an agent being unable to see past the one clause it was handed.

It partly solves two rules wanting to change the same sentence. Whether the combined edit is any good depends entirely on the lead agent, which is working from two short memos rather than everything the two agents actually figured out.

It doesn't solve edits that conflict in meaning without overlapping in text. Suppose one agent narrows the definition of "Confidential Information" in section 1, and a clause in section 11 was relying on the broad version. Those two edits don't touch the same words, so nothing flags them. The same goes for deleting a clause that something else cross-references. Only the final read-through could catch it, and that's one model call over a long document — not a systematic check.

This is the part I'd push hardest on in an interview. The mechanics are genuinely good; the handling of meaning-level consistency is thin.

“we gave each component of the document a unique identifier”

What does this mean concretely?

[Inferred] Concretely: when the contract is loaded, it gets broken into its structural pieces — sections, paragraphs, numbered clauses, list items, table cells, definitions, exhibits — and each piece is given a label that doesn't change, something like sec-8.2 or p-0147.

From then on, everything works in terms of those labels. The document is shown to the model with them attached, so it reads something like "[p-0147] Neither party shall assign this Agreement...". Tools take a label as their argument: read this section, propose a change to that paragraph. Citations are labels, so highlighting in the interface is exact. And checking whether two agents conflict becomes a matter of comparing lists of labels.

The reason this is worth doing is that the alternative — having the model quote text to say where it means — fails in three routine ways. Contracts repeat near-identical language, so "the Notices provision" might match three places. Models paraphrase slightly when quoting. And as edits accumulate, the original text an agent quoted stops matching what's in the document.

One thing that takes care: labels have to stay stable when the document changes. Inserting a new clause between 8.2 and 8.3 needs a fresh label rather than renumbering everything after it, which is why the internal identifiers are kept separate from the clause numbers a reader sees.

Does the orchestrator look at sub-agent output one at a time or all simultaneously in the same prompt?

[Post] Never stated. What we know: each agent finishes by filing "a short memo"; results are streamed as each agent completes; merging is described as working like version control, with only collisions going to the lead agent; and the lead agent "takes a final pass to validate the quality of the review as a whole."

[Inferred] It's almost certainly a mix of three things, and the word "short" in "short memo" is the clue.

Merging isn't a model call at all. It's code comparing which clauses each agent touched.

Conflict resolution happens one conflict at a time. Each one gets its own small model call containing just the original text, the two competing edits, and the two relevant memos. This keeps each call cheap and focused, and lets several conflicts be resolved at once.

The final validation has to see everything, because checking that every rule got handled and that the document reads consistently isn't something you can do in pieces. So the lead agent probably does see all the memos together at that point — and that is exactly why they have to be short. If each agent handed back its full working transcript, sixty of them wouldn't fit in one call.

That's the general shape of this kind of system: the agents doing the reading burn through enormous amounts of text, and only a compressed summary travels back up to the one that has to see the whole picture.

“Every agent in the hierarchy sees this context”

What does “see” mean here?

[Inferred] "Sees" almost certainly means it's written into every agent's prompt, rather than being something an agent has to go and ask for. Two things point that way: it's small and structured, just a handful of fields, and putting shared text at the start of every prompt is precisely what their caching optimization requires.

The likely exception is the precedent documents, which can be large. Those are more plausibly something an agent searches when it needs them than something pasted into sixty prompts.

Why this matters: it's the fix for one of the old system's most embarrassing failures, where reviews would "miss whose paper the contract was on." In a pipeline, that fact could be known at the classification step and simply absent by the time the redline step ran, because each step only got what the previous step passed along. Putting it in front of every agent makes it impossible to lose.

The cost is that you're paying for those tokens on every one of dozens of calls. That's exactly why prefix caching was worth a section of its own.

“persists with the review”

Any concerns about context window dilution?

[Inferred] The concern is right in general, but this design mostly avoids it, for three reasons.

Saved isn't the same as loaded. That state lives in a database attached to the review, not in a conversation that keeps growing. It gets pulled out selectively, which is what "we reuse the relevant agents" is telling you.

Each agent's world stays small. A subagent only ever sees one rule, the contract, and the deal context. If you ask a follow-up about rule 12, rule 12's agent comes back with rule 12's state — not the accumulated state of the other fifty-nine.

And what's saved is summaries, not transcripts. Classifications, chosen positions, edit summaries, reasoning. Not every search the agent ran and every clause it read along the way.

Where it does bite is the lead agent's final pass, which necessarily grows with the number of rules, and a long session where someone keeps iterating on the same review. The post doesn't say whether either of those gets trimmed.

“we reuse the relevant agents”

How is relevance determined?

[Inferred] Relevance is probably worked out structurally rather than by asking a model, and each of the three triggers has an obvious mechanical answer.

Switching the position on a rule is the easy one. You know which rule the user clicked on, so you re-run that rule's agent and nothing else.

A follow-up question sounds harder, but usually isn't, because the question is attached to a specific flagged issue in the interface. The user clicks on the redline and types their question, so you know which rule they mean without having to interpret the text.

The third case is the interesting one. When the user accepts some suggestions, the document changes, and any rule whose edits were aimed at the text that just changed is now working from a stale picture. Since every edit is already recorded against specific clauses, you can compute exactly which rules those are.

That's the second payoff from the identifier scheme — it was introduced to detect conflicts, and it happens to give you invalidation for free. The post doesn't claim they do this, but they've described every ingredient you'd need.

The risk classification scores in the eval results table seem very low. Any secondary review?

[Post] 59% rising to 77%, with no breakdown by category and no mention of any low-confidence re-check. The product is positioned as a first draft — "reliable first pass reviews," "giving users a well reasoned draft as a starting point." There is escalation to a human, but it's for situations where the deal has nuance the playbook can't capture, not for cases where the model is unsure.

[Inferred] Two separate questions here.

Is 77% actually low? You can't tell from the post, and that's the real problem. If it's measuring exact agreement with a lawyer's label on a three-way judgment call, 77% might be close to the ceiling — two competent lawyers won't agree 100% of the time on whether a deviation is acceptable or unacceptable, and the system can't beat a bar nobody has measured. Harvey never publishes how often their own lawyers agree with each other, which leaves the headline number floating without a reference point. That's the single biggest gap in the post.

Is it acceptable for the product? Yes, provided a lawyer reviews everything. A wrong label costs review time, not a bad contract — as long as the mistake is visible. The dangerous case is the clause the system never flagged at all, because nothing surfaces it for anyone to catch. That's the number worth asking about.

On secondary review specifically: the only second look described is the lead agent's final pass, and what that checks is coverage — that every rule got a verdict. Not whether the verdicts were right. An obvious addition would be re-checking the classifications the model was least sure about, or sending anything near the acceptable/unacceptable boundary to a second model. Nothing like that is mentioned.

“The issue would self-resolve usually on retry”

Why?

[Inferred] Because the failure isn't caused by the input, it's caused by the path the agent happened to take.

Model outputs are sampled rather than deterministic, so the same prompt produces a different first move on a different run. These loops almost always start with one bad early step — searching for a term that isn't in the document, misreading which party a clause binds — and then compound from there. Retrying re-rolls that first step.

There's a second effect that reinforces it. Once an agent has made a few failed attempts, those attempts are sitting in its own context, and the model is now predicting what comes next given a history of failure. It tends to produce more of the same. Starting over wipes that history, which is often more important than the re-roll itself.

And some fraction of "stuck" isn't the model at all — it's a slow or degraded response from the provider, which naturally clears.

Note the word "usually." Retrying doesn't rescue a genuinely hard clause, which is why they pair retries with timeouts rather than retrying forever. And retrying one rule without redoing the review is only possible because each agent works on its own copy.

“To mitigate this, we set timeouts by model, review phase, and document size”

What do timeouts in these different categories mean?

[Inferred] These are three inputs to a single time budget, and each one accounts for a different reason a step might legitimately take longer.

Different models work at very different speeds, especially those that do extended internal reasoning before answering. A single threshold would either cut off a slow model doing perfectly good work, or let a fast model hang for minutes before anyone noticed.

Different stages of the review have different natural durations. Classifying one clause against one rule is quick. Reconciling conflicts across dozens of memos is not. Holding both to the same limit means one of them is set wrong.

And bigger documents take longer for the obvious reason — more to read, more sections to search, more steps before the agent is satisfied. A 300-page contract can't be held to a 10-page contract's budget.

So the limit is looked up from those three facts, probably from a table calibrated against how long things actually took during their evaluation runs, and set well above the slowest legitimate case. The point isn't to make anything faster. It's to make sure that when one rule goes wrong, it costs you that rule and not the whole review.

“we added concurrency limits and exponential backoffs”

How to determine parameters of these?

[Inferred] These numbers aren't found by tuning the agent. They come from constraints outside it.

The cap on how many agents run at once is set by the rate limit Harvey has with the model provider, which is shared across everything Harvey does. You work backwards from it: take the tokens per minute you're allowed, divide by the tokens per minute a single agent consumes, and that's the most you can safely have in flight. Then you divide that budget among customers, so one person's 200-rule playbook doesn't consume the entire allowance and stall everyone else's reviews.

Backoff — waiting before retrying after being rate-limited — follows a standard recipe: wait about a second, double it each time, stop doubling somewhere around 30 to 60 seconds, and add a random amount to each wait. That last part is the one people forget and then regret. If forty agents all get rejected at the same instant and all wait exactly two seconds, they all retry at the same instant and get rejected again. Randomizing the wait spreads them out.

The actual tuning is empirical: raise the concurrency cap until you start seeing rejections and total review time stops improving, then settle just below that point.

One interaction the post doesn't address: time an agent spends waiting in the queue shouldn't count against its timeout. If it does, heavy load causes timeouts, which cause retries, which cause more load.

“We used that fact to redesign our prompts to optimize for prefix cache hits”

How to design prompt to enable this?

[Inferred] First, what the caching actually does. When you send a prompt, the provider does expensive setup work on it before generating anything. If the beginning of your prompt is identical to one it processed recently, it reuses that work — charging a fraction of the usual price and skipping the delay. The catch is that it only reuses the part that matches from the very first character, and it stops at the first difference. One changed word near the top wipes out the benefit for everything below it.

That gives you the design rules.

Put the most shared content first and the most varying content last. The order that works is: system instructions, then tool descriptions, then deal context, then the contract, then the specific rule, then the specific task. Any layout that mixes rule-specific text into the contract destroys the shared portion.

Make the shared part character-for-character identical across every agent. No timestamps, no request IDs, no agent number in a header, no data structures that serialize their fields in a different order each time.

And within a single agent's back-and-forth, only ever append to the conversation. Rewriting an earlier turn invalidates everything after it.

The payoff here is unusually large because the contract dominates the prompt and every single agent reads the same one. That's why one change improved both cost and speed.

“we enabled streaming results such that each rule’s result becomes available as soon as its subagent finishes”

Any concerns about quality given lack of orchestrator review at this stage?

[Inferred] Yes, and this is the one place where the post's own argument works against itself. The reason the lead agent exists is that raw output from individual agents can be contradictory. Streaming shows the user exactly that raw output, before anyone has reconciled it.

Three things can go wrong. A user reads rule 7's suggestion, and then reconciliation rewrites it to accommodate rule 23 — so they have to read it again, and may have already made a decision based on the version that's now gone. If they can accept a suggestion before reconciliation finishes, they can accept one half of a pair of edits that were supposed to be resolved together. And watching answers change on screen makes a system feel unreliable even when every change is the system doing its job correctly.

The standard ways to handle this: stream the classifications early, since reconciliation rarely changes whether a clause is a problem, but hold the actual edits until they've been merged. Or show streamed edits marked as still in progress and don't let anyone accept one until the review completes. Harvey's mention of exposing progress suggests something along these lines, but they don't say whether a rule that's about to be reconciled is visibly marked as such.

“Our evaluation showed that no single model performed best across subtasks of contract review”

Does subtasks mean rule classification vs redlining here or is it more granular where different rules get different models?

[Post] By stage of the review, not by rule: "We built the system to switch models across phases of the same review, selecting each one based on its measured quality, latency, and cost for that specific task."

[Inferred] The stages visible in the post are classification, searching and navigating the document, drafting redlines, resolving conflicts, and the final validation pass. These genuinely do want different things. Resolving a conflict between two competing edits is hard reasoning over a small amount of text, which favors the most capable model available. Navigating a 200-page contract to find the assignment provisions is easy reasoning over a lot of text, done many times, which favors whichever model is fastest and cheapest.

[Search] This lines up with how Harvey talks about its model lineup publicly, which sorts models by exactly that kind of profile: Claude Opus 4.6 for "agentic reasoning and deep multi-step legal analysis," Sonnet 4.6 for "strong performance with a significantly lower latency footprint," Gemini 3 Flash for "where speed and throughput matter." That's a by-stage taxonomy, not a by-rule one.

[Inferred] Sending harder rules to stronger models would be a sensible next step, but nothing in the post claims they do it, and it needs something the post never describes: a way to predict which rules are hard before you've reviewed them. More in Q37.

“We built our agent harness to normalize the differences between model providers.”

What does this mean concretely?

[Inferred] It's a layer of their own code sitting between the agent logic and each provider's API, so that the agents are written once against Harvey's own interface rather than three times against three different ones.

The differences it has to absorb are more than cosmetic. The request and response formats differ, including where the system instructions go and what the streaming events look like. Tool calling differs — what the mechanism is called, how tool schemas are written, whether the model can request several tools at once, what happens when it produces malformed arguments. The controls for how much the model thinks before answering differ and don't map cleanly onto each other. Caching differs: one provider wants you to mark explicitly where the cacheable part of the prompt ends, another does it automatically, so a prompt laid out efficiently for one isn't necessarily efficient on the other. And the operational details differ throughout — how rate limits are signalled, what the error types are, how tokens are counted.

This matters more for Harvey than for a typical chat product because their entire scalability story assumes models are interchangeable parts. Picking the best model per stage, setting timeouts per model, backing off per provider — none of that is manageable if each model is its own integration.

The part a harness doesn't solve is that prompts aren't portable. A prompt tuned around one model's habits often does worse on another. So you get the ability to swap models mechanically, but you still have to re-run your evaluations every time you do.

“Every accepted, edited, or rejected redline helps agents understand a legal team's preferences.”

In what way is this feedback actually incorporated?

[Post] It isn't yet. Note the tense: "This will allow us to build towards a personalized system, from users to accounts." No mechanism is described.

[Inferred] There are four ways to do this, and they differ a lot in cost and in how soon you could ship them.

The cheapest and almost certainly first is to use past decisions as examples. Store which redlines the team accepted and which they rejected, and when an agent is about to work on a given rule, pull up how that team handled the same rule before and put it in the prompt. No training required, works per customer, and it uses the same infrastructure as the historical-contract search in Q35.

The second is to improve the playbook rather than the model. If a team rejects your liability fallback every single time, that's not a model problem — the playbook is wrong. Surface it as a proposed change to the rule for a lawyer to approve. This keeps a human in the loop and improves the artifact the customer can actually read and audit.

The third is to train a small model on the accept/reject history and use it to filter out suggestions the team is unlikely to want before they ever appear.

The fourth is fine-tuning the main model on preferences. This is the most powerful and the least likely soon: per-customer fine-tuning doesn't scale, fine-tuning on everyone's data averages away the personalization that was the point, and one customer's contracts can't be used to improve another's reviews.

The complication running through all of these is that a rejection doesn't tell you why. A lawyer might reject a technically correct redline because the counterparty is important and this isn't the fight to pick, or accept a mediocre one because they're behind on the deal. Without capturing the reason, the signal stays noisy — which is probably part of why this is still future work.

“This opens the ability to search across thousands of documents for previous examples to trust agent outputs and quickly understand what a team has approved in the past and why.”

Who is doing this searching and in what part of the pipeline?

[Inferred] The sentence actually describes two different users doing two different things at two different points.

One is the agent, during the review. Before deciding how to handle a provision, it searches the company's past contracts for how that provision was settled before, and lets that shape which position it pushes for. This changes the output, and it's the personalization path.

The other is the lawyer, after the review. That's what "to trust agent outputs" points at: the interface shows a suggestion next to a note saying your team has agreed to this exact language fourteen times before. This doesn't change the suggestion at all, only whether the lawyer believes it.

My guess is they ship the second one first, because the worst case is much milder. A bad retrieval there is a weak citation the lawyer ignores. A bad retrieval in the first case is a bad redline that goes out the door. Both need the same underlying thing: an index of the company's signed contracts broken down by provision, recording what language was finally agreed and whether it was negotiated.

Worth noting what changes between today and that future. Right now the user chooses which precedent documents matter. Once the system chooses, retrieval quality becomes a new way for the review to be wrong.

How would you improve this system if you were building it today?

In rough order of value.

Fix the measurement before touching the architecture. Publish how often two lawyers agree with each other on the same clause, so 77% has something to be compared against. Report how often the system catches the unacceptable deviations specifically, since that's the expensive miss. And have lawyers who didn't write the rubrics grade a held-out set blind. Right now the headline numbers can't be interpreted, and every decision about what to build next rests on them.

Detect conflicts in meaning, not just in text. Their merge catches two edits touching the same words. It can't catch an edit that narrows a definition in section 1 when section 11 depends on the broad version. When the contract is parsed, build a map of which clauses define terms, which clauses use them, and which clauses cross-reference which. Then check the merged document against that map in code. It's a deterministic check, not a model call, and it closes the gap that worries me most.

Group related rules instead of one agent per rule. Four rules about limitation of liability currently produce four agents that each read the same section, each work out the same thing, and then collide. Cluster rules by which part of the contract they target and give one agent the cluster. That removes duplicated work and removes the conflicts before they happen rather than resolving them afterward.

Escalate when the model is uncertain, not only when the deal is unusual. Today escalation is about deal nuance. Add a confidence score to each classification and route the uncertain ones — especially anything sitting near the line between acceptable and unacceptable — to a stronger model or to a person, marked as needing judgment. This is the cheapest way to reduce the risk of a silent miss.

Route by rule difficulty as well as by stage. They pick a model per stage of the review, but difficulty varies far more between rules than between stages. A rough difficulty signal — how long the rule is, how many conditions it has, how many sections it touches — would let the easy majority go to a fast model and concentrate the budget on the hard minority.

Start the feedback loop now, as retrieval. Accept/reject history is the most valuable thing they'll accumulate, and you don't need to train anything to use it. Index it per customer and feed relevant past decisions to the agent as examples. And capture a reason when a lawyer rejects something, even a one-click reason, or the data stays ambiguous forever.

Make streaming honest. Either hold edits back until they've been reconciled, or show them marked as provisional and don't allow accepting one until the review finishes. Otherwise streaming quietly gives away the consistency guarantee the lead agent exists to provide.

Is the main difference between prompts and context that context is information + instructions and prompts are just instructions?

Yes, pretty much. Context is also often dynamic.

“we need strategies for managing the entire context state (system instructions, tools, Model Context Protocol (MCP), external data, message history, etc).“

In what form is MCP included in the context?

MCP isn't a protocol object in the window — it gets flattened into ordinary text. Three forms, MECE by when they land:

1. Connection time — tool schemas. Each server advertises tools; the client renders each as name + description + JSON Schema, injected into the tool-definitions block. This is the dominant cost: tens to hundreds of tokens per tool, paid every turn. Twenty servers × 15 tools is easily 20k+ tokens before you say anything.
2. Connection time — server instructions, resources, prompt templates. Servers can return an instructions string (typically appended near the system prompt) plus listings of available resources and prompt templates.
3. Call time — tool results. Whatever content blocks the server returns land in the message history and persist there.

Point 1 is why MCP makes the post's "bloated tool sets" failure mode the default rather than a mistake you have to commit.

“This results in n² pairwise relationships for n tokens.”

Is this true for modern architectures as well? What about local attention?

True of the mathematical object, increasingly false of the implementation. Three things to keep apart:

1. Compute vs. memory. FlashAttention removed the n² memory cost (it never materializes the attention matrix) but keeps n² FLOPs. GQA and MLA shrink the KV cache, not the pairwise count.
2. Sparse and local attention genuinely break n². Sliding-window attention (Mistral, Gemma) costs n·w. Frontier models interleave — a few global layers among many local ones. DeepSeek's NSA learns which blocks to attend to. [Search] 2026 work is predominantly hybrid: sparse + linear + a few full-attention layers.
3. It doesn't rescue the conclusion — it strengthens it. A token in a sliding-window layer cannot see distant tokens directly; it depends on multi-hop propagation up the layer stack. So long-range degradation is, if anything, more likely under local attention. The post's "attention budget" claim survives even though its stated mechanism is a simplification of what frontier models actually run.

“Additionally, models develop their attention patterns from training data distributions where shorter sequences are typically more common than longer ones.”

Don’t models typically get fine-tuned on representative data lengths even if pre-training data has more shorter sequences?

[Post] Doesn't address it — just asserts the pretraining-distribution claim.

[Inferred] Your objection is correct and the post's framing is dated. The standard modern recipe is:

1. Pretrain short (4k–8k) for most of the token budget — cheap.
2. A long-context extension phase: continued pretraining at 32k → 128k → 1M, using RoPE base frequency scaling (ABF) or YaRN, on data deliberately upsampled for long documents and synthetic long-range tasks.
3. Long-context SFT / RL on tasks requiring long-range dependence.

Three things blunt the correction:

1. Volume. The long-context phase is typically low single-digit percent of total training tokens. Parameters are still overwhelmingly shaped by short-sequence gradients.
2. Data quality. Genuine documents with real long-range dependencies are scarce. Much long-context training signal is constructed — concatenated documents, synthetic needle tasks — which teaches retrieval more than reasoning across the whole span.
3. Measurement. Needle-in-a-haystack is near-saturated, but multi-hop and aggregation-over-long-context benchmarks (RULER and its successors) still degrade sharply with length.

Net: the post's mechanism is stated too simply; the empirical claim holds for reasons that survive your correction.

What’s a two sentence summary of position encoding interpolation?

Rather than asking the model to handle position values it has never seen, you rescale the positions of a longer sequence down into the range it was trained on — position 8,000 in a 32k input is fed as if it were position 2,000 in the original 8k range. This works because interpolating inside the trained range is far safer than extrapolating outside it, but it compresses the positional signal so adjacent tokens become harder to distinguish, which is the "degradation in token position understanding" the post mentions.

“smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome”

Dual objective optimization or satisficing?

[Post] Written as if it were a joint optimum, and never resolved. The post does give itself an escape hatch elsewhere: "minimal does not necessarily mean short."

[Inferred] As literally written it's ill-posed — you cannot minimize tokens and maximize success without a stated exchange rate. Three readings:

1. Scalarized dual objective. max P(success) − λ·tokens. Coherent, but λ is never stated, and in practice λ is tiny next to the cost of a task failure.
2. Constrained. min tokens subject to P(success) ≥ threshold. This is what "smallest possible set" implies grammatically, and it is the defensible reading.
3. Satisficing, in practice. Nobody maps the frontier. Teams reach "good enough," trim context until quality visibly drops, and stop.

[Inferred] The more useful reframe: it is effectively lexicographic — quality first, tokens as tiebreaker — because over the relevant range the two objectives are not opposed. Adding low-signal tokens hurts both. A real tradeoff only appears once every remaining token is genuinely informative, which most production prompts never reach.

“System prompts should be extremely clear and use simple, direct language that presents ideas at the right altitude for the agent.”

How good are LLMs at writing system prompts for other LLMs? Could I give an LLM examples of good and bad system prompts and expect it to come up with a good system prompt for my system?

[Inferred] Split into two capabilities, because the answer differs sharply:

1. From examples of good/bad prompts alone — weak. Style transfer works; you'll get something that looks like a good prompt. But prompt quality isn't a property of the text, it's a property of text × task × model. With no task data the model has no signal about your failure modes. Expect a plausible first draft, not a good prompt.
2. From examples of good/bad task outputs plus a metric — strong. This is the real answer, and it is a solved-enough tooling problem. [Search] The two current standards are DSPy's MIPROv2 (bootstraps demonstrations, proposes candidate instructions, Bayesian search over combinations) and GEPA (reflective evolutionary search — the model reads execution traces and failures, then mutates the prompt). GEPA generally wins at lower rollout budgets. TextGrad and OPRO are the earlier academic line.

[Inferred] Concrete path, matching the post's own recipe: hand-write a minimal prompt → build a 30–100 case eval set with a scorer → run GEPA or MIPROv2 against it. The eval set is the work; the optimizer is cheap. Without evals you are doing vibes whether a human or an LLM writes it.

“then add clear instructions and examples to improve performance based on failure modes found during initial testing”

How to address risk of overfitting?

[Post] Never mentions the risk. This is a real gap — as written, the recipe is textbook dev-set overfitting.

[Inferred] Mitigations, MECE by where you intervene:

1. Data. Hold out a test set you never inspect during iteration. Refresh the dev set from production traffic periodically. Keep a regression suite of already-fixed cases to catch whack-a-mole.
2. Edit policy. Prefer general fixes to specific ones — this is the post's laundry-list-vs-canonical-examples point restated as bias-variance. A rule naming a specific input is high-variance; a rule naming a principle generalizes. If you cannot articulate the principle behind a patch, that is the signal you are fitting a sample.
3. Budget. Cap prompt growth. Every N iterations, delete the oldest instructions and re-run evals — most will not be load-bearing. Prompts accrete monotonically unless you force a deletion step.
4. Model. Re-test on a different or newer model. Instructions that help only one model are usually compensating for that model's quirk, not encoding your task.

“One of the most common failure modes we see is bloated tool sets that cover too much functionality or lead to ambiguous decision points about which tool to use.”

How can you detect and mitigate this issue?

Detection, MECE by signal type:

1. Static — no traffic needed. Embed tool descriptions and flag high-similarity pairs. Run the post's heuristic as an actual exercise: two engineers independently label which tool each of ~50 realistic requests should use, then measure agreement.
2. Behavioral — from traces. Per-tool call rate (never-called tools pay token rent every turn for nothing); tool error and retry rate; thrash patterns where tool A is followed by tool B on the same subgoal; definitions as a share of total context.
3. Interventional. Ablate: remove a tool, re-run evals. The only signal that is causal.

Mitigation, MECE by action:

1. Merge near-duplicates into one tool with an enum parameter.
2. Delete anything below a usage floor.
3. Tier — a small always-loaded core set, plus dynamic loading of the rest (tool search, or per-task MCP server loading).
4. Disambiguate in the description — state explicitly when not to use it and which tool to use instead. Cheapest fix, usually the highest yield.

“However, teams will often stuff a laundry list of edge cases into a prompt in an attempt to articulate every possible rule the LLM should follow for a particular task. We do not recommend this. Instead, we recommend working to curate a set of diverse, canonical examples that effectively portray the expected behavior of the agent.”

One person’s laundry list is another person’s diverse set. How to distinguish?

[Post] No criterion offered. Genuine soft spot in the post.

[Inferred] The distinction is real and testable. Three tests, cheapest first:

1. Generalization test. Does this example teach a rule applicable to inputs you haven't written down? Canonical examples cover a region of the behavior space; laundry-list items are point masses covering only themselves. If removing an item breaks only the exact case it names, it is laundry.
2. Coverage-vs-count test. Map examples against the task's dimensions (input type × difficulty × output form). A diverse set is few examples spread wide. A laundry list is many examples clustered in one corner — usually whatever broke most recently.
3. Ablation test. Remove each item, re-run evals. Decisive, and the only empirical one.

[Inferred] Two fast heuristics that usually settle it without running any of the above:

• Provenance. Laundry lists grow append-only from incident response; canonical sets are curated top-down from a task taxonomy. If your examples section is a log of past bugs, it is a laundry list regardless of how varied it looks.
• Form. Laundry-list items are exceptions ("if X, don't do Y"). Canonical examples are demonstrations (full input → output pairs).

“agents built with the “just in time” approach maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools”

What does it mean to “maintain lightweight identifiers”? Where and how?

[Inferred] "Maintain" covers three different storage locations, MECE by where the identifier sits:

1. In the context window, as plain text. The agent ran ls or grep -l and the paths are now in the message history. That is the maintenance — a path costs ~10 tokens, the file ~10,000. No machinery required. This is most of what Claude Code does.
2. In the environment, re-derived on demand. The identifiers are not held at all; the filesystem is the index, and the agent regenerates paths with glob/grep whenever needed. Zero context cost, one round trip.
3. In external persistent storage. A NOTES.md, a to-do list, or the memory-tool file system the post introduces later. The durable form — and the only one that survives compaction.

[Inferred] The honest answer: this is mostly not a data structure you build. It is a discipline enforced in your tool contracts — make search return paths + line numbers + a two-line snippet rather than file contents, and the identifiers maintain themselves. The engineering work is in tool return shapes, not in an identifier registry.

“primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time”

These feel like very rudimentary ways to access relevant files? Is lexical similarity actually sufficient?

[Inferred] For code, the objection is weaker than it looks, for three reasons:

1. Code isn't prose. Identifiers are exact, near-unique strings. Want UserAuthService? Grep it — every definition and call site, perfect precision. Embeddings are worse here: they blur exact tokens, which is the opposite of what you want.
2. It's agentic, not single-shot. The comparison is not one grep vs. one embedding query. It is one embedding query vs. fifteen greps with a model reading results and reformulating between each. The loop supplies the semantics; grep supplies the lookup. Recall misses get recovered in later iterations.
3. Freshness and zero infrastructure. No index to build, invalidate, or sync. On a repo the user is actively editing, a stale index is worse than none.

[Inferred] Where it genuinely fails: conceptual queries with no shared vocabulary — "where do we handle rate limiting?" when the code calls it ThrottleGuard. The model's world knowledge partly covers this (guess synonyms, grep again) but not reliably. That is the case for hybrid: semantic search over docstrings or file summaries to find the neighborhood, then grep for precision inside it.

“model to summarize and compress the most critical details”

How does the model know what to include in the summary and what the most important details are? Do you just ask it to figure that out itself or is it more sophisticated than that?

[Post] Answers more than you would expect. It names the categories — the model "preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages" — and says engineers should be "carefully tuning your prompt on complex agent traces." So: it is a prompt, and the categories come from the engineer, not the model.

[Inferred] But it is more structured than "summarize this":

1. Schema-driven, not free-form. Production compaction prompts specify an output skeleton — goal, decisions made, current state, open problems, files touched, next step. The model fills slots. Far more reliable than open-ended summarization.
2. Deterministic pre-filtering runs first. Tool result clearing (the post's own example, now a Claude Developer Platform feature) involves no model judgment at all. Cheap deletions happen before expensive summarization.
3. Recency passthrough. Recent turns are copied verbatim rather than summarized. The five-files detail is this same idea applied to files.

[Inferred] What is not happening: there is no training signal on "did this summary preserve what mattered." Importance is hand-specified and tuned offline against traces — which is exactly why the post frames it as a recall-then-precision prompt engineering exercise rather than a learned component.

“The agent can then continue with this compressed context plus the five most recently accessed files”

Why five? What happens if the files are very big?

[Inferred] On five — almost certainly an empirical constant, not a derived one:

• It is a recency heuristic proxying for relevance. In coding, files you just touched are the ones you are about to touch. Cheap and usually right.
• Five is roughly where marginal value crosses marginal token cost. At ~500–2,000 tokens per source file, five is ~2,500–10,000 tokens — low single-digit percent of a 200k window. Twenty files would be a meaningful fraction of a freshly cleared window.
• It is the kind of number tuned once on internal traces and then frozen.

On big files — the more interesting question, and the post ducks it. Three possibilities:

1. A token budget binds, not the count. Any sane implementation caps total tokens, so "five files" is really "up to five, subject to a budget." Otherwise one vendored 500k-line file blows the fresh window immediately.
2. Truncation or re-read-on-demand. Either partial inclusion, or keep only the path and let the agent re-read the relevant span with offset/limit — which is precisely the just-in-time pattern the post argues for earlier.
3. If neither: compaction becomes self-defeating — you summarize to free space, then immediately refill it.

[Inferred] Worth flagging the tension: re-injecting file contents is pre-loading, which contradicts the post's own just-in-time thesis. Re-injecting paths would be the consistent choice. The likely justification for pre-loading anyway: after a reset the agent has lost the reasoning that made those files relevant, so paying tokens to restore working state beats a slow rediscovery loop.

“Start by maximizing recall to ensure your compaction prompt captures every relevant piece of information from the trace, then iterate to improve precision by eliminating superfluous content.”

How would recall and precision actually be measured in this situation? Or is it more qualitative?

[Inferred] Both are measurable, but only once you fix a "unit of information" — which is the whole difficulty. Three levels, MECE by cost:

1. Qualitative human review — what most teams actually do. An engineer reads a long trace, lists what matters, and checks the summary against it. Recall = fraction of that list present. Subjective and slow, but catches the failures that matter.
2. Proposition-level with an LLM judge — the practical middle. Decompose the trace into atomic claims ("the auth bug is unfixed", "we chose Postgres over Mongo"). A judge checks each against the summary. Recall = claims preserved / claims extracted. Precision = summary claims supported by the trace / total summary claims — note this catches hallucination, not just verbosity.
3. Downstream task accuracy — the only ground truth. Generate question-answer pairs from the full trace, then ask them of an agent given only the summary. Recall failure = questions it can no longer answer. This measures the thing you actually care about (can the agent continue) and is the standard formulation in long-context and memory benchmarks.

[Inferred] One precision note: the post's "precision" is not the classification-theoretic one. It means token efficiency — no superfluous content. So the operational metric is closer to compression ratio at fixed recall.

“the agent regularly writes notes persisted to memory outside of the context window”

1. How does the model know what to include in the notes? Is it application-dependent?
2. Why written to memory outside of context window? How does it know when to retrieve?

(1) What to include

[Post] Says both things, in tension. Engineered on one hand (Claude Code's to-do list, a NOTES.md convention); emergent on the other — the Pokémon agent, "without any prompting about memory structure," develops maps of explored regions, tracks achievements, and maintains combat strategy notes.

[Inferred] Reconciliation: the trigger and affordance are engineered (a tool exists, the prompt says to use it); the schema is often emergent. And yes, strongly application-dependent — what to persist is whatever must survive a reset in that domain: architectural decisions for coding, entity relationships for research, map/inventory/strategy state for a game.

(2) Why outside the window, and when to retrieve

Why outside: [Post] "provides persistent memory with minimal overhead" — notes survive compaction and resets; in-context content by definition does not.

[Inferred] A second reason the post does not give: notes are rewritable. Context is append-only — you cannot revise an earlier message, only append a correction, and both then sit in the window. A file can be edited in place, so stale state gets replaced instead of accumulating alongside its correction.

When to retrieve: [Post] Doesn't say, beyond "these notes get pulled back into the context window at later times" and "after context resets, the agent reads its own notes."

[Inferred] Three mechanisms in practice, MECE by who decides:

1. Unconditional on reset — the post-compaction prompt says "read your notes file."
2. Model-decided — the memory tool is a filesystem; the agent reads when it judges it needs to. Unreliable without prompting.
3. Always-on injection — small files (a to-do list) get re-rendered into context every turn.

[Inferred] The real gap: retrieval timing is the weakest link in the whole scheme. An agent that does not know it forgot something has no trigger to go look.

“After context resets” What does reset mean here?

[Post] Undefined. Used once, in the Pokémon section, next to "coherence across summarization steps" — implying reset ≈ compaction event.

[Inferred] It conflates three things worth keeping separate, MECE by what survives in-band:

1. Compaction. The window is replaced by a summary of itself. Some information survives in-band. This is what the post means in that sentence.
2. Hard clear. The window is wiped to system prompt plus a fresh start; nothing survives in-band, only external files carry over. This is what "reset" most literally means, and what a game harness does when it hits the limit.
3. New session. A separate process started later — the same information situation as (2) but across time. This is what the memory tool targets: "maintain project state across sessions."

[Inferred] Why the distinction matters: the note-taking argument is only load-bearing for (2) and (3). For (1), a well-tuned compaction prompt can carry state in-band. Notes still win because they are lossless for what they cover and do not degrade across repeated resets — a summary of a summary of a summary drifts; a file does not.

How would you improve this system if you were building it today?

Ordered by expected value:

1. Make context management measurable, not advisory. The post's entire toolkit is heuristic with no evaluation story. Build a continuation benchmark: sample real traces, compact them, then measure whether the agent can answer trace-derived questions and complete the task. That converts "tune your compaction prompt" into a regression test.
2. Tiered editable state instead of one summarization pass. Compaction today is monolithic and lossy in a single shot, and re-summarizing a summary compounds drift. Better: verbatim recent turns, plus a structured state document (decisions, open problems) that is edited rather than re-summarized, plus cold storage that is searchable but unloaded.
3. Treat tool definitions as retrievable context. MCP made bloated tool sets the default. Load a small core set and retrieve the rest on demand via tool search. This directly attacks the failure mode the post names but does not solve.
4. Give note-taking an index. Per Q16, retrieval timing is the weak link. Keep a cheap always-in-context table of contents of what is in memory, so the agent knows that something exists before deciding whether to read it. Knowing you forgot is the precondition for looking.
5. Reconcile just-in-time with the five-files behavior. Post-compaction, re-inject file paths plus one-line summaries, not contents. Cheaper and consistent with the rest of the argument.
6. Measure the write side, not just the read side. Everything in the post is about what enters context. Nobody measures what the agent writes — note quality, note staleness, contradictory notes accumulating. That is where long-running agents actually fail.

“customer-reported misses in Detection 360”

1. What is Detection 360?
2. How do customers realize misses happened?

(1)

An in-product submission channel: "Submit a missed attack or a false positive incident from within the product's Detection 360 tab" (navigation: Investigate → Detection 360). The case object carries sender, recipient, subject, submitted-by, case number, status (Submitted/Resolved), and a VIP flag. The platform page summarizes the loop: "Detection 360 investigates, explains the outcome, updates the models." Case lifecycle runs "from initial classification through investigation, remediation, and deployed detections." Turnaround is marketed as "hours, not days," and the loop reportedly got "94% faster."

(2)

Abnormal only ever uses analyst-initiated framing ("An analyst reports a vendor impersonation attack"). Their AI Security Mailbox "connects to existing phishing-report buttons, abuse mailboxes, G Suite alerts, and third-party tools" — but no page states that end-user reports automatically create Detection 360 submissions. They are documented as separate paths.

[Inferred] The realistic channels, MECE by who notices:

1. End user reports it — phishing button or abuse mailbox, then a SOC analyst escalates
2. SOC proactive hunting — searching mail for indicators pulled from threat intelligence
3. A different tool catches it downstream — EDR fires on the payload, web proxy blocks the link, a SEG flags a sibling message
4. Post-incident, working backwards — someone clicked, credentials were phished, money moved
5. External notification — the impersonated vendor, a partner, or law enforcement
6. Purple team or phishing simulation — a deliberate test that got through

Where does this agentic pipeline fit into the broader system? Given latency constraints

It is clearly offline and asynchronous. The post gives four tells:

• It is triggered by a submission, not by an arriving message
• It evaluates "across a larger slice of the customer's real traffic" — historical replay
• It loops "multiple times"
• End-to-end is "hours"

So the pipeline sits beside the detection engine, not inside it. Same architecture as the 2024 rules post: expensive LLM work happens once per attack pattern offline; a cheap deterministic artifact runs online.

The three-tier picture:

[Inferred] This places the agentic loop in the middle speed tier — faster than retraining, far slower than a scoring decision. Which is exactly the gap the 2024 post identified: "adding new [features] involves retraining the model, which takes weeks." The agents fill that hole. That is the one-sentence answer to how it fits.

“how they normally communicate”

In what form?

Sender-history profile, computed per sender-recipient pair and per sender-tenant:

• Has this sender written to this recipient before, and how often
• Typical send times and cadence
• Typical infrastructure — sending IPs, domains, authentication posture
• Typical recipients within the organization
• Typical subject/topic distribution and tone
• Signature blocks and formatting conventions

Form: almost certainly a structured summary — counts, rates, first-seen dates — plus probably a handful of retrieved prior messages for the agent to read.

“how the message compares to typical organizational patterns”

In what form?

• Does this organization receive Teams event invites at all? From which platforms?
• Which vendors are established relationships versus first contact?
• What is the normal distribution of external senders, of financial-request language, of invite mail?

“what signals point toward intent or impersonation”

In what form?

[Post] Not specified, but the example output enumerates a real subset: sender domain and its subdomain lineage, authentication results (SPF/DKIM), body link destinations, subject content, delivery mechanism. [Post] elsewhere: "thousands of rich signals."

[Inferred] The likely full set adds display-name-vs-address mismatch, first-time-sender flags, phone numbers extracted from the body, brand mentions inconsistent with sender identity, and the text-model scores from the NLP post (urgency, tone).

[Inferred] Form: most likely a text serialization of the feature record — name/value pairs with short natural-language annotations — because the agent's output reasons over them in prose. Standard pattern: render structured features into a prompt, let the model reason, give it tools to query more.

[Inferred] The design question worth raising: with "thousands" of signals you cannot render them all into a prompt. So something selects. Either the agent has tools to query attributes on demand — which the post hints at, "its tools let it verify those intuitions against real data" — or a retrieval step pre-selects. On-demand querying is the better design and is what the post implies. Same feature-selection-as-retrieval problem flagged in Q4 on the 2024 post; it recurs here.

“This is a **trusted platform abuse** attack - using Microsoft Teams' event registration system to deliver phishing content.“

If the Deep Research Agent already determined it’s an attack, what’s the Adaptive Detection Agent doing?

[Post] The Deep Research Agent's output is "a detailed behavioral profile of the attack" — prose, explaining why this one message is malicious.

[Post] The Adaptive Detection Agent's job is to "find the attributes that represent the underlying behavior behind the attack," check statistical and semantic relevance, apply second-order abstraction, test against traffic, and iterate.

Three ways to state the difference:

1. Explanation vs. classifier. The DRA answers "why is this message malicious?" The ADA answers "what decision rule separates this class of message from all normal traffic?" A perfect explanation may cite features that do not generalize, or that are not even available as production attributes.
2. Prose vs. an executable artifact grounded in the production feature space. The DRA can say "uses legitimate Microsoft Teams infrastructure." The ADA must translate that into attributes the engine actually computes at scale, in sub-second, and verify each one separates on real traffic. The DRA has no obligation to check whether "uses Teams infrastructure" also fires on ten thousand legitimate meeting invites a day. The ADA does.
3. One message vs. a population. The DRA reasons about a single known-bad message with zero false-positive cost. The ADA operates against a one-in-a-million FP budget over a traffic distribution.

[Inferred] The analogy: the Deep Research Agent writes the incident post-mortem; the Adaptive Detection Agent writes the regression test. Knowing exactly why the bug happened does not hand you the test that catches the next one.

[Inferred] One more asymmetry: the DRA's determination is retrospective, on a message already known to be a miss. The ADA must produce something that works prospectively, on messages nobody has labeled.

“The problem is that without any deeper understanding, teams end up with rules that are technically correct but practically useless. A system might flag that 100% of attacks in a cluster lack an attachment, which is true but meaningless since most legitimate emails don't have attachments either. Statistical relevance alone produces a lot of noise without the next critical piece”

This seems like it’s only true for the most rudimentary definition of statistical relevance?

Yes. The critique is correct, and this is the strongest pushback available on this post.

Their example is a statement about P(no attachment | attack) — recall on the positive class alone, with no reference to the negative class. Every standard criterion rejects it immediately:

So what they describe is not "statistical relevance" — it is support on the positive class. A first-year decision-tree splitting criterion catches this. They picked a weak example and undersold their own argument.

[Inferred] The real case for semantics is different, and stronger. Three genuine reasons statistics alone is insufficient here:

1. Multiple comparisons at tiny n. A submission yields maybe a handful of attack messages against thousands of candidate attributes. At small n and high dimensionality, many attributes separate perfectly by chance. Statistics cannot distinguish causal from coincidental at that sample size. Semantics functions as a prior over hypotheses that prunes the search space before testing — exactly what you need when the multiple-testing burden is otherwise fatal. Note this is a well-posed statistical argument, not an anti-statistical one.
2. Durability under adversarial shift. Statistical relevance is measured on the past and says nothing about which correlations survive the attacker's next move. "Sender domain equals X" is maximally statistically relevant and minimally durable. Semantics is how you predict stability — their own second-order-thinking point, and the argument they should have led with.
3. Cost asymmetry is not in the statistic. At a base rate under 1 in 10 million and an FP target of 1 in a million, an attribute with excellent in-sample lift can still be catastrophic at production volume. Understanding what an attribute means is how you anticipate where it fires outside your sample.

[Inferred] Accurate framing: semantics is a prior and a durability filter, not a substitute for statistics.

[Inferred] The charitable reading, in their defense: at n of about 5 attack messages, most rigorous statistical criteria are genuinely underpowered — you cannot estimate lift reliably from five examples. So their practical situation may justify leaning hard on priors. That is probably what they meant; the attachment example just does not communicate it.

“It has enough statistical sense”

How? Reasoning, tool-use or something else?

[Inferred] Tool use, not reasoning. The LLM does not compute statistics in-context. It calls something that queries traffic and returns counts and rates, then reasons over the returned numbers. That is the only design that works, for three reasons:

1. LLMs are unreliable at arithmetic over large numbers, and these are counts over millions of messages
2. The data is not in the context window — it is in their feature store and data lake
3. The post explicitly describes an iterate-and-verify loop, which requires an external oracle

[Inferred] The likely tool surface: "for attribute A and recipient R, what fraction of the attack messages have value v, and what fraction of normal traffic does?" plus "run this candidate detector against this traffic slice and return the hits."

The "statistical sense" that actually lives in the model is choosing which query to run and interpreting the result — not computing it.

[Search] Consistent with their broader practice: their backtesting infrastructure already does exactly this kind of replay, "re-hydrated exactly as they would have been during online scoring."

[Inferred] Worth naming the pattern: LLM-as-orchestrator over a statistical oracle. Same division of labor as the 2024 post — LLM does translation and hypothesis generation, deterministic software does evaluation. The same team making the same architectural choice twice, two years apart, is a real signal about their design philosophy.

“The second-order approach is to ask: what's true about the domain that makes it suspicious? Is it a domain the organization has never seen before? Is it newly registered? Those characteristics will still hold when the attacker switches to a different domain next time.”

Is the Adaptive agent simply trying to answer yes or no to these questions, or use them in its downstream determination in the same iteration as well?

[Post] answers this indirectly: "its tools let it verify those intuitions against real data before committing to a rule." So the sequence is hypothesize, verify, commit — and the verified abstraction becomes a predicate in the rule, not a fact that is consulted and discarded.

[Post] The end state confirms it: "The result is detectors that generalize well beyond the specific messages they were built from." That only holds if the abstraction (domain novelty, domain age) is what the shipped detector tests, not the raw domain.

[Inferred] The flow for the PayPal example:

1. Observe — all the attacks share sender domain *.teams-events.com
2. Hypothesize the second-order property — "a domain this organization has never seen before," and/or "recently registered"
3. Verify against traffic — does never_seen_domain actually separate these attacks from this recipient's normal mail?
4. Commit — if yes, the abstraction goes into the detector; the raw domain does not

So: both. It is a hypothesis that gets answered yes/no and, if yes, becomes part of the rule in that same iteration. The yes/no is the verification gate; the attribute is the payload.

[Inferred] Worth naming: this is generate-and-test over a hypothesis space — classic program synthesis with an execution oracle. The LLM proposes candidate abstractions; traffic data accepts or rejects them.

“they latch onto the most distinctive surface features of the example they were shown”

What does distinctive mean here?

[Inferred] Precisely, it means maximally identifying of the specific example, relative to the corpus prior — the lowest-probability, highest-information strings in the input. For the PayPal message that is the exact domain jorjarounsevell.s08.usa1.teams-events.com, the exact phone number, the exact subject line.

Three framings of why models gravitate there — same phenomenon, different vocabulary:

1. Information-theoretic. A rare string carries high self-information. Asked "what characterizes this message," the rare strings are the highest-signal answer to that question as literally posed.
2. Learning-theoretic. From n of 1, the minimum-description-length hypothesis that perfectly separates the example is the example itself. Memorization is the optimal fit. Generalization requires a prior preferring abstract hypotheses, and nothing supplies that prior unless you put it there deliberately.
3. Shortcut learning. The model takes the feature with the highest in-sample discriminative power at the lowest reasoning cost. Exact string matching is both.

[Inferred] The essential point: distinctive is measured on the example; predictive must be measured on the class. Those coincide only if the example is representative in the right way — and for an adversary-generated example they systematically do not, because the attacker controls the distinctive features precisely so they can rotate them.

So in this domain, "most distinctive" and "most durable" are close to anti-correlated. That inversion is the entire reason second-order thinking has to be engineered in rather than emerging naturally.

“We worked to guide the agent toward behavioral abstractions over raw values”

By prompting, fine-tuning or what?

[Inferred] The wording points away from fine-tuning and toward three softer levers, in order of likely impact:

1. Attribute vocabulary design — probably the biggest lever, and structural rather than persuasive. If the attributes exposed to the agent are already behavioral — never_seen_sender, from_fqdn_age_in_days, both real names from their 2024 DSL — then a first-order rule becomes hard to express. You do not have to talk a model out of matching a raw domain if no attribute matches raw domains. This is guidance by construction, and it is far more reliable than instruction.

2. Prompting. Explicit instruction plus few-shot examples contrasting first-order and second-order rules for the same attack. Cheap — and "we worked to guide" / "something we spent a lot of time tweaking" is precisely what iterating on a prompt sounds like.

3. The verification loop as an implicit filter. A first-order rule passes local eval but generalizes badly in the broad eval — it will not catch other instances of the attack type. [Post] treats that as an explicit signal: "Did it find other instances of this attack type that were previously missed? If so, that's a strong signal." So the evaluation selects for generalization even when the model's first instinct is first-order. Guidance via reward rather than instruction.

[Inferred] Why probably not fine-tuning: you would need a labeled corpus of good-vs-bad detectors, which is exactly the scarce thing this system exists to produce; "guide" and "tweaking" are not fine-tuning vocabulary; and [Search] their documented fine-tuning work (7B Mistral/Llama-2 via LoRA) is described for detection and classification, not agent behavior.

“First, the agent runs a fast local evaluation. It takes the attack messages it already identified along with a random sample of normal traffic for the recipient and tests the detector against them.”

The random sample would need to be huge for this to work, right?

Yes — and the post effectively concedes it, twice.

[Post] The local stage is explicitly not a precision estimate: "This is intentionally lightweight. The goal is to iterate quickly." And the broad stage exists because the sample is insufficient: "it lets us catch any false positives hiding in the broader traffic that didn't show up in the smaller sample."

So the local sample's job is screening, not estimation — kill obviously broken candidates cheaply before paying for a full replay.

[Inferred] The arithmetic, worth having ready. Observing zero false positives on a sample of size n bounds the true rate only weakly. By the rule of three, with zero events in n trials the upper bound of the 95% confidence interval is about 3/n:

A "random sample of normal traffic for the recipient" — one mailbox, days or weeks of mail, so hundreds to low thousands of messages — is three to four orders of magnitude short of supporting a precision claim. It is entirely adequate for what it actually does.

[Inferred] The second, subtler problem: the sample is drawn from the wrong distribution. Randomly sampling one recipient's normal mail gives mostly easy negatives — internal mail, known vendors, newsletters. The false positives that matter are near misses: legitimate Teams invites, real PayPal notices, genuine first-time vendor contact. Random sampling surfaces those only in proportion to their frequency, which is low. So you systematically fail to detect the exact failure mode the detector is most likely to have.

[Inferred] The fix: stratified hard-negative sampling. The Deep Research Agent has already characterized the attack as "Teams event registration abuse." Use that to sample negatives conditioned on the characterization — real Teams event invites, real event-platform mail, real PayPal correspondence. Small, targeted, far more informative per message than a random draw. Standard hard-negative mining, and it fits their architecture cleanly since the characterization is already in hand.

“Every message the detector would have flagged gets reviewed, and we LLM-label each one to determine what it actually is.”

If we’re using an LLM to label it again, wouldn’t that just lead to artificially inflated agreement?

Good instinct, and partly right — but the failure mode is narrower than self-grading.

First, why it is not circular in the obvious way: [Post] the detector is not an LLM. It is a rule over attributes; the LLM wrote it offline. So at evaluation time you have a symbolic classifier being graded by a language model — different mechanisms, not a model scoring its own forward pass. That defuses the strongest version of the objection.

[Inferred] But three genuine correlation risks remain, MECE by where the leak enters:

1. Shared priors between writer and grader. If the same model family that reasoned "Teams invite plus urgency equals attack" also labels flagged messages, it will tend to agree — not because the detector is right, but because both inherit the same notion of what phishing looks like. Legitimate mail that merely looks phishy to that model gets labeled malicious, and a real false positive is scored as a true positive.
2. Context contamination. If the labeler sees the detector's rationale or the Deep Research Agent's attack profile, it is being primed with the hypothesis it is supposed to test. Strongest version of the problem, and the most easily avoided.
3. No ground-truth anchor. [Post] gives no indication that LLM labels are ever calibrated against human labels. Without that you cannot distinguish "the detector is precise" from "the labeler agrees with the detector." The number is uninterpretable in isolation.

[Inferred] What to do, in order of return:

• Blind the labeler — it sees the message and nothing about the detector or the attack profile. Cheap, eliminates risk 2 outright.
• Use a different model family for labeling than for generation — decorrelates errors. Same logic as their multi-tokenizer ensemble in the NLP post, applied to judges.
• Anchor on a human-labeled calibration set — periodically measure the labeler's own precision and recall against expert labels, then report detector precision corrected for labeler error.
• Human review on the tail — auto-accept where the labeler is confident and volume impact is small; escalate high-volume or VIP-recipient detectors.

[Search] They already know how to do this. Their 2024 post Testing GenAI Products describes using "a separate judge LLM, with access to our accuracy rubric" and running "separate 'unrestricted' red team LLMs." The judge-LLM discipline is established internally — this 2026 post just does not say whether those safeguards apply here.

“The agent loops through this cycle multiple times, refining the detector with each pass until it can reliably catch the attack without compromising on safe emails.”

Stopping criteria?

[Inferred] What a real implementation needs, MECE by termination reason:

1. Success — precision on the broad eval above a threshold, and recall on the submitted attacks at or near 100%, and firing volume within budget. All three, not just the first.
2. Budget exhaustion — max iterations, or max LLM/eval spend. Necessary because the loop is not guaranteed to converge.
3. Plateau — no improvement over k consecutive iterations. Cheaper than burning the full budget.
4. Abandonment — declare the attack not separable with the available attributes and escalate to a human. [Post] never mentions this possibility, and it must exist. Some attacks genuinely cannot be caught by a rule over current attributes; the honest outcome is "this needs a new feature" or "this needs the model," not a marginal rule. A loop with no failure exit will eventually ship something that satisfies the letter of the criteria and not the spirit.

[Inferred] The real risk the post never names: eval-set overfitting through iteration.

Every loop uses evaluation results to modify the detector. That is adaptive data analysis — after k rounds the measured precision on that traffic slice is optimistically biased, and the bias grows with k. You are effectively doing gradient descent on your test set.

Mitigations, strongest first:

• A final held-out gate — a fresh traffic slice never seen during iteration, evaluated exactly once. Fail there and the detector does not ship, regardless of how good the iteration numbers looked.
• Fresh slice per iteration, so no single slice gets fit repeatedly.
• Penalize iteration count — a detector that needed 12 rounds is more suspect than one that passed in 2, because it has absorbed more evaluation information.

This connects to Q32: the two-stage eval already has the right shape; it needs a third, sacrosanct stage that iteration never touches.

“Every submission kicks off the full loop”

What is a submission? Does the loop get kicked off by a single false negative?

[Search] A submission is a Detection 360 case, filed in-product: "Submit a missed attack or a false positive incident from within the product's Detection 360 tab." The case object carries sender, recipient, subject, submitted-by, case number, status, and a VIP flag.

So yes — a single customer-reported message can start the loop. Note it can be a false positive as well as a false negative, though this post only describes the missed-attack path.

[Inferred] But the loop does not stay at n of 1, and the post quietly tells you so. It says the agent tests against "the attack messages it already identified" — plural. That means between submission and generation, one reported message gets expanded into a cluster: the Deep Research Agent (or something upstream) finds related messages in the customer's traffic. This has to happen, because you cannot do statistical attribute selection on a single example.

[Inferred] Two consequences:

1. Cluster quality gates everything downstream. Expand too narrowly and you are back to n of 1, with no evidence to verify the second-order abstraction against. Expand too broadly and you are characterizing a mixture, so the detector ends up loose or incoherent. The post says nothing about how this expansion works, and it is arguably the most load-bearing undocumented step in the pipeline.
2. A single submission is an unvalidated trigger. [Search] No public Abnormal source addresses adversarial or poisoned submissions. An attacker with access to a tenant's Detection 360 — or who can get a legitimate message reported as an attack — could induce the system to auto-generate and deploy a detector that fires on legitimate traffic. [Post] There is explicitly no human in the loop: "No analyst queue, no manual review, no waiting." The evaluation stages are the only defense, and they are graded by an LLM (see Q33).

How would you improve this system if you were building it today?

Stage 1 — Understand the attack

• Fix the input bias, or at least measure it. Detection 360 misses are the misses somebody noticed, not a random sample (Q21). Instrument it: periodically sample random traffic for expert labeling to estimate the unreported miss rate, so you know how much of the problem the loop can even see. Without this you cannot tell whether "hours to protection" covers 10% or 90% of your misses.
• Document and harden cluster expansion (Q36). Report cluster size and purity alongside every generated detector, at minimum.

Stage 2 — Generate the detector

• Enforce second-order by construction, not by prompting (Q31). Expose only behavioral attributes, or type-check candidate rules and reject raw-literal predicates. Guidance that lives in the schema cannot be talked out of; guidance that lives in a prompt can.
• Generate a population, not a single detector. Sample K candidates, evaluate all, keep the precision/recall Pareto front. More LLM calls, but calls are cheap relative to a bad detector.

Stage 3 — Evaluation

• Hard-negative sampling in the local stage (Q32), conditioned on the attack characterization the Deep Research Agent already produced.
• Blind the LLM labeler, use a different model family, add a human-labeled calibration set (Q33) so labeler error can be corrected for rather than ignored.
• A held-out final gate the iteration loop never touches (Q35). The change to argue hardest for — without it, the reported precision is optimistically biased by construction.

Stage 4 — Precision optimization

• Explicit stopping and abandonment criteria (Q35), including the option to fail and escalate rather than ship a marginal detector.
• Penalize iteration count as a proxy for overfitting risk.

Stage 5 — Deployment

• Blast-radius controls. Auto-deploy in score-contributing mode first; promote to verdict-forcing after a live observation window. Cap how much a newly generated detector can quarantine per hour. Auto-rollback on FP-rate breach.
• Tiered human review. Fully automatic for low-volume, low-impact detectors; mandatory review above a volume or VIP-recipient threshold. Preserves the "hours, not days" claim for the common case while capping tail risk — and that framing is how to sell it internally without gutting the product story.

Cross-cutting

• Lifecycle and decay. Same gap as the 2024 post, but worse here, because generation is now fast enough to accumulate rules quickly. Per-detector monitoring (fire rate, precision, and marginal catch — messages it catches that nothing else does), automatic demotion when a detector stops catching, expiry by default.
• Reconcile per-tenant vs. global (Q21). There should be an explicit promotion path: a tenant-local detector that proves out across multiple tenants gets promoted to global, at a higher evidence bar.
• Submission trust model (Q36). Provenance checks, rate limits, and a poisoning red-team exercise. [Search] They already run red teams for prompt injection on their GenAI products; this loop deserves the same and there is no public evidence it gets it.
• Feed detectors back into the model. Detectors are a fast patch for a slow retrain. Mine the accumulated detector population for recurring attribute combinations — those are missing features for the core model. Over time the detector layer should shrink as the model absorbs what it keeps rediscovering. Without that, you are building an ever-growing rule layer and calling it progress.

Do different payment processors send different kinds of data? If yes, would this affect the approach you use?

Yes. Ramp likely has a "normalization" or "ETL" (Extract, Transform, Load) layer that maps various processor formats into a standardized internal schema before the AI agent ever sees it.

What format do they use for the input to the LLM?

Not stated in the blogpost but probably something along the following lines:

1. System Prompt with Guardrails
2. Transaction Metadata, User Request Data, Retrieved merchants, specified actions provided as a structured list like JSON or Markdown

Why use an LLM instead of a simpler model?

Simpler models

1. can’t provide explanations as easily
2. are harder to use for multimodal and unstructured data
3. would struggle with zero-shot because they don’t have world knowledge from pre-training

“By bringing in merchants using these two strategies, the LLM can inspect the names, websites, and categories of a good selection of merchants. It can then decide whether to create a new merchant or modify existing ones.”

It doesn’t look like they’ve mentioned how to produce a final list for the LLM. What’s our best guess for how they did the final ranking from the two lists?

Simple approach

1. Take k/2 from each list to get k total results

Cross-Encoder Re-ranker

1. Pull n >> k from each list to get 2n total results
2. Use a cross-encoder (using all the features you have for both the transaction and the merchant) to rerank and filter for the final k
3. Have LLM pick from this final list

Did they use any kind of few-shot prompting? If not, is that likely to have been helpful?

Unclear from the blogpost whether they used it.

It could be helpful, especially for edge cases like the government/Service Fee stuff and making it clear to the LLM what kind of explanation you want to provide users. Examples of both correct and incorrect responses could be helpful.

“We require that the LLM always chooses one of the provided actions”

It doesn’t look like they’ve mentioned how they actually enforce this restriction. What’s our best guess for how they did this?

They say “If the LLM hallucinates, we inform it of its mistake and have it retry until we get a valid response.” so it looks like they have retry logic to help with this.

They could also use Structured Output (potentially through Constrained Decoding).

“If the LLM hallucinates, we inform it of its mistake and have it retry until we get a valid response.”

What guardrails could they have put in place to avoid getting stuck in infinite loops?

1. Limit number of retries
2. Dynamic temperature adjustment
3. Meta-Prompting for Errors (provide explanation of failure to decrease likelihood of future failures in retries).

Once you hit the maximum number of retries without a valid response, you can escalate to humans.

“We take our primary LLM's rejection reasoning and then use a second LLM to rewrite it in plain language.”

Is the second LLM there because the primary LLMs rejection reasoning is unlikely to be very accessible to users? If yes, how could they have set up the second LLM to address that gap?

Yes, it’s better to exclude the human-optimized explanations from the primary LLM to avoid overloading it and keep it focused on its main job.

The second LLM can be prompted with something like

“You are a helpful Ramp Customer Support agent. Your goal is to explain a technical rejection to a user in a friendly, clear, and non-technical way. Do not use terms like 'MCC' or 'LLM' or 'Card Acceptor.' Use the provided evidence to explain the 'Why' simply.”

“It is also important to recognize LLMs' knowledge that has been distilled as part of their training.”

If this is the case, would it be helpful to replace the LLM every so often to incorporate this new knowledge? Or would a better strategy be to just store all instances of name changes and use a mapping between old names and new names?

Leveraging the User-Provided Data is probably a better way to update the LLM’s knowledge. You only need to do it once for each merchant whose name changed.

Storing mappings is complicated. Multiple companies can have the same name, only distinguishable using metadata like location, industry etc. Probably not worth it.

They mention “multimodal retrieval augmented generation (RAG)”.

Where is the multimodal part used?

Related receipt images; could use a Vision-Language Model (VLM) or a specialized OCR-to-text pipeline

How is the LLM as a judge supposed to know if the “agent's LLM action resulted in an improvement according to the user's request”?

The Judge is likely using a Rubric-Based Evaluation. The prompt for the Judge LLM for changes could look like this:

"Compare the Original Merchant with the New Merchant created by the Agent.

1. Does the New Merchant better match the Receipt Evidence (Line items: Fuel)?
2. Does the New Merchant fulfill the User's Intent (Requested: Gas Station)?
3. Is the New Merchant a valid entity (not a hallucination)?

Decision: If 'Yes' to all, mark as IMPROVEMENT. If the agent rejected the request, check if the User's input was 'Placeholder' or 'Malicious'—if so, mark it as REASONABLE REJECTION."

They said using LLM as a judge “Requires Cycles”; what does this mean?

Not sure what “Cycles” refers to here but I guess the general idea is that it requires a lot more work to set up. Maybe some degree of iteration on the prompt, input data etc to get it working correctly.

What is the “Mean Daily Improvement Rate on Classification Correction Requests”?

The 98.8% figure means that on a typical day, the Judge finds that 98.8% of the agent's changes resulted in a more accurate merchant record than what existed before. It is essentially an "Accuracy Score" for the actions the AI chooses to execute.

The reasonable rejection rate seems really low (one-third of legitimate user feedback is being ignored)? Should they be concerned about this?

Probably a worthwhile risk given that accepting a bad user request will mess up the database.

How would you improve this system if you were building it today?

1. Agentic Search for relevant merchants
   1. An agent could look at the merchant’s website, social media or Google Maps page to understand it better. Addresses from receipts can also be reverse-engineered to find Google Maps pages which can then provide a lot of additional useful data.
2. Speculative Decoding or Router Architecture (easy requests handled by fast model, harder requests handled by better model) to speed up responses
3. Consider adding some LLM as a judge negatives to the few-shot examples in the prompt

“citations were attached to the cell as a whole”

Does “attached to the cell” here mean that the citation mapping was (full generated text) <> (source document)? Or was it (sentence in generated text) <> (source document)?

I think they mean that the original was (full generated text) ↔ (source document) and they moved to (sentence in generated text) ↔ (source document).

“Previously, review tables generated sources through an algorithm that combined model-generated text and fuzzy matching.”

What would this methodology might have looked like concretely?

This could potentially look something like the following:

1. Generation step: The model produces a free-form answer for the cell (e.g., a summary of a contract clause) without any grounding mechanism built into generation itself.
2. Post-hoc matching step: Separately, the system takes that generated text (or key phrases from it) and runs a fuzzy string-matching algorithm (e.g., something like Levenshtein distance, n-gram overlap, or approximate substring search) against the source document to find the closest matching passage.
3. Attach citation to cell: Whatever passage scored as the best fuzzy match gets attached as "the citation" for that cell — but since matching happens on the whole generated text rather than per-sentence, the granularity is coarse

How can citations at a sentence level be generated?

This isn't explicitly outlined in the text some potential ways include:

1. Prompting-based: The model could be instructed via system/prompt engineering to output structured citations (e.g., "for each sentence, output the corresponding source index") as part of its generation — essentially asking the model to self-report indices as a formatting/output requirement.
2. Constrained decoding / structured output: Rather than relying on the model to "remember" to cite correctly via prompting alone, the system could constrain the model's output format at the token level (e.g., forcing it to emit index tokens at sentence boundaries), which is more reliable than prompting alone but requires deeper integration with the inference pipeline.
3. Retrieval-augmented indexing: The source document could be pre-chunked and indexed (e.g., by sentence, with unique IDs), and the model's job is to select from a known, indexed candidate set of sentences rather than generate a citation from scratch — reducing hallucinated citations by construction.
4. Hybrid: Prompting to elicit reasoning + a separate verification/alignment step that checks whether the model's claimed index actually matches the claim.

Sharing the reasoning with users seems kind of obvious. Is there any reason why they didn't do this in the previous iteration?

1. Latency/cost?
2. Risk of hallucination?

“This also resulted in a boost in output speed. By reassessing our execution pipeline, we were able to adopt an implementation that kept per-cell latency low and the overall architecture simple while achieving sentence-level granularity. At scale — a 30-column, 1000-document table produces 30,000 concurrent cells — the latency of this approach is substantial.“

I don't understand why this new approach is faster. Where is the latency boost coming from exactly?

They don’t explain it in the blogpost but it could be because:

1. Parallelization: processing citation-lookup across sentences concurrently rather than sequentially per cell
2. Reduced post-processing: eliminating the separate fuzzy-matching pass (which requires comparing generated text against source text as a distinct step) by having the model point to indices directly during generation — removing an entire post-hoc computation stage

Let's say I've built this pipeline and I want to get applied legal researchers' resources to assist in the evaluation. How can I, in advance of securing these resources, estimate how much manpower I need? One approach I had in mind was using synthetic queries in LLM as a judge to get a preliminary estimate of the improvement, and then use that to calculate the sample size I need for statistical significance at a desired margin of error.

This is probably reasonable but we can enhance this by estimating p̂ (preliminary win-rate) on a much smaller sample (30-50) which we can then use to calibrate the LLM judge by computing the agreement rate.

“Reliability measures the rate at which Harvey fails to provide the user with an answer for parsing reasons or because of unexpected model outputs.”

How do you measure “unexpected model outputs”?

Not explained in the blog post, but could feasibly include:

1. Schema/format violations — the model was expected to return output in a structured format (e.g., answer + reasoning + citation index), but instead returned malformed structure, missing fields, or extra unparseable text.
2. Invalid citation indices — the model points to a document index/offset that doesn't exist or is out of bounds, which would fail downstream parsing even if the "answer" text itself looks fine.
3. Refusals or non-answers — model declines to respond, or returns something like an apology/meta-commentary instead of the expected output type.
4. Truncated or incomplete generations — output cut off mid-structure (e.g., due to token limits) such that required fields never complete.
5. Encoding/formatting anomalies — unexpected characters, broken JSON, or malformed markup if the pipeline expects a specific serialization.

Most of these can probably be measured/detected by LLM as a judge or some simple heuristics.

“We measured quality through side-by-side preference testing.”

Is there any concern about just comparing the two of them and asking which one is better, as opposed to quantifying the degree of improvement? If I want to justify the significant resources dedicated to this project, the current setup (which doesn’t distinguish between narrow wins and big wins) seems like it makes it a bit difficult.

We could try a graded preference scale: "A much better," "A slightly better," "Tie," "B slightly better," "B much better." This is still a comparative judgment (avoids the calibration problems of absolute scoring across raters) but recovers magnitude information that pure binary throws away.

How would I convince a business leader that this project was worth doing?

I think some way to get a sense of the time-savings from the improvements would be helpful (maybe ask ALRs to estimate).

If we’ve had previous user research showing that latency is a driver of churn, reduced latency can be presented as a major win. It’s fair to assume that improved response quality (as detected by the ALR preference data) is good for the bottom-line (increased retention, positive word-of-mouth etc.).

How would you improve this system if you were building it today?

1. Citation verifications step (latency and cost trade-off though)
2. More structured reasoning trace
3. Selective reasoning depth depending on what kind of cell output is being considered
4. Task-based outcome metrics in evaluation (actual time-to-complete a review task, error/override rates in production)
5. Failure handling: if the primary pipeline produces an unparseable or low-confidence output, retry with a simpler/more constrained generation mode (e.g., force extraction-only, no reasoning) rather than surfacing a hard failure to the user
6. Tiered latency budgets by query complexity. Not all cells need the same processing depth; routing simple yes/no extractions through a faster path and reserving the full reasoning+citation pipeline for interpretive questions would reduce average latency without sacrificing quality where it matters

How does query understanding fit into the broader search pipeline? What are the remaining stages?

Piecing together Instacart’s other engineering posts, the pipeline is roughly:

1. Query understanding — normalization/spell correction, category classification, rewrites, SRL tagging
2. Retrieval/recall — hybrid: lexical (Postgres full-text search, GIN indexes, a customized ts_rank), embedding-based retrieval (their ITEMS bi-encoder over a FAISS ANN service), and category-based retrieval
3. Merge/fusion — reciprocal rank fusion or a weighted convex combination of lexical and semantic scores
4. Ranking — multi-objective, balancing “relevance, popularity, and personal preference”; the embedding score is a feature here
5. Serving layer — ads insertion, store-level availability filtering, dedup, blending, pagination
6. Post-serving — out-of-stock replacement suggestions, feedback logging

What kinds of “traditional machine learning models” have historically been used for search-related tasks at Instacart for the different use-cases they outlined?

[Post] Only two are named, both in the Fig 1 caption: query classification “relied on a FastText model for multi-label classification,” and query rewrites “were generated by a separate system that mined user session behavior” — which is a mining/statistics pipeline, not really a model.

[Post] For category classification the legacy setup is described as a “massive multi-class classification problem” predicting “the top-K most likely categories from a flat list.”

[Search] Adjacent Instacart posts name ITEMS (Instacart Transformer-based Embedding Model for Search), a Sentence-Transformers bi-encoder fine-tuned on their conversion data, and an earlier MiniLM-L3-v2 bi-encoder used for hybrid retrieval. Both are retrieval-side, not QU.

[Inferred] SRL is never given a legacy model. The pre-LLM standard for e-commerce query tagging is BIO sequence labeling — CRF, BiLSTM-CRF, or BERT token classification. Reasonable guess, unstated in the post.

Why would the methods they've historically used struggle with broad queries?

• No lexical bridge. FastText is a bag-of-n-grams linear classifier over the surface string. “healthy food” shares no characters with Yogurt, Produce, or Whole Grains. The model can only memorize the association from conversion data — it can’t derive it.
• The label distribution is genuinely flat and multi-modal. Top-K with a fixed threshold either cuts real categories (low recall) or admits noise (low precision). There’s no single correct answer to fit toward.
• It learns popularity, not semantics. Conversions for a broad query are dominated by whatever is merchandised or best-selling, so the model converges on “what people buy” rather than “what the query means.”
• No compositional reasoning. “healthy” is a constraint over attributes (low sodium, high fiber, no added sugar), not a category. Deriving that requires world knowledge the model doesn’t have.

“QU operates upstream and doesn’t benefit from direct feedback like clicks or conversions. The pseudo-labels we derive from user behaviors are inherently noisy.”

Wouldn't this be a problem at any layer of the stack, regardless of what methods you use?

It’s a matter of degree rather than kind — but the degree is large enough to change what’s possible:

• Ranking gets directly attributable labels. An item was shown at position 3 and clicked. The label sits on the same object the model scores.
• QU’s outputs are never shown to a user. A conversion is credit-assigned through retrieval → ranking → UI. If someone searches “bread” and buys bananas, you can’t separate a bad rewrite from bad retrieval from bad ranking from a user who simply changed their mind. That’s credit assignment through a long, non-differentiable pipeline.
• Confounding is worse upstream. QU’s own output determines the candidate set, so the feedback you observe is censored by the thing you’re trying to evaluate. Ranking has a version of this — position bias, selection bias from retrieval — which is why IPS and click models exist. But ranking has a label to debias; QU only has a proxy.
• QU labels are set-valued and partly subjective. “What are the correct categories for ‘healthy food’?” has no unique answer, which is why the fallback is human evaluation — expensive, and the reason the post reaches for LLMs at all.

“This heterogeneity introduced inconsistencies”

What kinds of inconsistencies?

The realistic failure modes:

• Semantic disagreement between components. The classifier maps “gluten free bread” to Bakery while the rewriter expands it to “sourdough” — which contains gluten. Two parts of QU assert contradictory things about the same query.
• Taxonomy version skew. Each model trained against a different snapshot; categories get renamed, split, or merged, so one model emits IDs another considers dead.
• Preprocessing skew. Different tokenization, casing, stemming, and spell correction per system, so “zip-lock” and “ziplock” are treated identically by one component and differently by another.
• Training-window skew. Rewrites mined over 30 days, classifier trained over 90 — different implicit definitions of “popular.”
• Metric incomparability. One team’s precision is human-labeled, another’s is a conversion proxy. You can’t reason about end-to-end quality because the numbers don’t compose.
• Training/serving skew duplicated N times, each with its own bugs.

“slowed down development cycles”

Why?

• N systems × (data pipeline + training + eval + serving + monitoring) is N times the maintenance and on-call burden, with no shared leverage.
• A change to a shared input — a taxonomy revision, a new catalog field — forces N retrains and N separate validations.
• Each system needs its own labeled eval set, and building one is the long pole. Every new task starts from zero.
• No transfer. Improving the rewriter’s handling of “organic” does nothing for the classifier. Effort doesn’t compound.
• Launching a new QU capability means a new bespoke model — a multi-quarter project — versus writing a prompt.

“Trained on diverse textual data, LLMs possess world knowledge that enables them to make logical inferences from user queries”

Isn't this also true for models like BERT, which I'm assuming were part of their historical pipeline?

Yes, BERT has world knowledge — but three differences are load-bearing:

• Scale. BERT-base is 110 million parameters trained on roughly 3 billion words. A frontier LLM is three to four orders of magnitude larger on both axes. “Italian parsley ≈ flat parsley” is exactly the kind of long-tail fact that only survives at that scale.
• Access mode — the decisive one. BERT’s knowledge lives in its representations, and the only way to extract it is a trained head on your labeled data. If you have no clean labels (Q4), you cannot get at it. An LLM exposes the same knowledge through natural-language instructions, requiring zero labels. That’s the whole unlock.
• Generation and steerability. Rewrites are a generative task — BERT can tag spans but cannot produce “curly parsley” as a substitute; it has no decoder over that space. And instruction-following is what makes “generate a broader query, not a synonym” expressible at all. That single capability is what makes the three-rewrite-type design in section 2 possible.

Short version: BERT has world knowledge. It doesn’t have label-free, generative, steerable world knowledge.

“We build data pipelines that retrieve and inject Instacart-specific context”

What are good ways to determine what context would be most helpful for a given task?

A practical method, in order:

1. Build an error taxonomy first. Sample 100–200 failures and label why each was wrong. Missing-knowledge errors (the model doesn’t know “verdant machine” is a brand) → context helps. Reasoning or format errors → context won’t help; prompting or fine-tuning will. This one step prevents most wasted effort.
2. The human-oracle test. Could a smart new hire with no Instacart data access answer this? If not, what would they look up? That lookup is your retrieval source. Cheapest, highest-signal step available.
3. Leave-one-out ablation. Build context as independent blocks — conversions, catalog, taxonomy, session — and ablate each against a fixed eval set. Keep only what moves the metric; every block costs tokens, latency, and adds distraction risk.
4. Prefer high-precision, low-volume context. Top-5 converted categories beats top-50. Irrelevant context measurably degrades LLM output, so more is not better.
5. Check for label leakage. If your context essentially contains the answer for head queries, your eval overstates gains on the tail — precisely where you need it to work.
6. Make sure it works cold. The tail queries you care about have no conversion history by definition. Include at least one source that works with zero engagement data — which is exactly why Instacart pairs conversion history with catalog embedding similarity.

“For the most advanced use cases, we fine-tune models on proprietary data”

Is there a good way to know in advance or in a less resource-intensive way whether fine-tuning is likely to be helpful for your use case before you dedicate the full set of resources needed for it?

Cheapest-first sequence:

1. The ceiling test. Can a frontier model with your best prompt and context solve the task well? If no, fine-tuning a smaller model almost certainly won’t — fine-tuning transfers behavior, it doesn’t invent capability. If yes, you have a teacher, and the problem reduces to a known-feasible engineering exercise. Instacart is exactly this case, which is why their story works.
2. Name what fine-tuning is for. Three distinct goals with very different success rates: (a) cost/latency — distill a known-good big model; high success. (b) format and style adherence — very high success, small data. (c) new knowledge — low success; RAG is usually the better tool.
3. Measure the few-shot slope. Run 0-shot → 5-shot → 30-shot. A steep improvement means the model learns this task from demonstrations, and fine-tuning is just many more demonstrations. A flat curve means the bottleneck is knowledge or capability, and more examples won’t fix it.
4. Run a small LoRA pilot. 1,000–3,000 examples, a few GPU-hours. The curve from 500 → 1,000 → 3,000 tells you whether more data pays. Hours, not quarters.
5. Check data feasibility before compute feasibility. Can you produce 5,000–10,000 high-quality examples without human labeling? If not, the project’s real cost is annotation, and that’s what usually kills it.
6. Check task stability. If the label definition churns monthly (taxonomy revisions), fine-tuning bakes in something you’ll repeatedly have to re-bake. Prefer RAG.

“Our legacy approach treated this as a massive multi-class classification problem”

Is every element of every level of the taxonomy hierarchy a class in this model? Would something be able to classified as both “Meat” and “Short Ribs”?

[Post] Yes to both, and the post’s own example proves it: for “butter milk” the legacy model predicts (“Dairy”, 0.95) and (“Milk”, 0.92) — explicitly described as “distinct, non-hierarchical outputs.” So the label space is flat and spans levels, and an ancestor plus its descendant can both be returned, because the model has no representation of the hierarchy.

Terminology flag: the post is internally inconsistent. The body says “multi-class”; the Fig 1 caption says “a FastText model for multi-label classification.” The behavior described — top-K, multiple simultaneous labels — is multi-label. Multi-class means exactly one label.

[Inferred] The pathology this creates: independent labels mean probability mass splits across redundant categories, the top-K budget gets spent on duplicates, and you get no signal about how specific to be. The new approach fixes this by predicting a category path (“the LLM’s predicted category path”), which is hierarchy-aware by construction — one output encodes both the level and the lineage.

“It directly powers recall and ranking, helping us retrieve items from the right categories and intelligently expand the search when a query is broad or ambiguous”

1. By “right categories”, do they mean they apply filtering?
2. How does search expansion work?

(a)

[Search] Instacart’s embeddings post lists “category-based retrieval” as a retrieval source alongside keyword and embedding retrieval.

[Inferred] Almost certainly not a hard filter in the general case. Two likelier uses:

• As a candidate generator — predicted categories trigger an additional retrieval pass (“fetch popular items in category C”), whose results merge with lexical and embedding candidates.
• As a ranking feature — item’s category is in the predicted set → boost.

Hard filtering on a QU prediction is dangerous: one classification error zeroes out recall for the entire query. Filtering is plausible only for high-confidence narrow queries, or in combination with the similarity guardrail.

(b) [Inferred] Expansion here means widening the candidate pool across taxonomy branches, not rewriting the text. For “healthy food,” many categories score highly, so you retrieve from each and blend — which is also what produces category-shelf-style result pages for broad queries. Note this is a different mechanism from the query rewriting in section 2, even though both get called “expansion.”

“First, being trained on noisy conversion data (e.g., a user searches “bread” but buys bananas) means it can produce irrelevant suggestions”

Shouldn’t the noise average out and thus be mitigated?

Averaging only rescues you if the noise is zero-mean and independent of the label. Neither holds here:

• The noise is systematic, not random. “bread → bananas” happens because users do multi-item shops in one session. The mislabeling correlates with what’s popular overall — bananas are among Instacart’s top items. So every query accumulates bias toward the same popular categories. More data makes that bias more confident, not smaller. This is the crux: you’re not averaging away noise, you’re converging on a biased estimate.
• There’s a feedback loop. The current system determines what’s shown → what converts → tomorrow’s labels. The noise is self-reinforcing, not i.i.d.
• Volume asymmetry. Averaging needs samples. Head queries have plenty, so it looks fine there. Tail queries — the entire target of this project — have a handful of conversions, so variance never washes out. This is exactly why the post pairs “noisy labels” with “data sparsity” as separate but compounding problems.
• Precision is the operative metric. Averaging shrinks mean error but doesn’t remove the tail of visible failures. A small persistent probability mass on a wrong category still surfaces to a user as a visibly bad result.

“we retrieve the top-K converted categories for each query as initial candidates”

Wouldn’t this struggle with low cardinality queries? Do they do any kind of rewriting or normalization to mitigate that?

[Post] Not addressed. No normalization or backoff step is described on the classification path.

[Inferred] This is a genuine gap, and arguably the weakest seam in the section. If a query has three historical conversions, the candidate set is tiny and possibly wrong — and a re-ranker cannot recover a category that was never in the candidate list. The LLM’s world knowledge is wasted if the retrieval step already threw away the right answer.

Standard mitigations, all of which Instacart plausibly has but doesn’t mention:

• Back off to a rewritten/normalized form. They already generate synonyms and broader queries — retrieve conversions for those instead. The post never connects the two systems, but it’s the obvious composition.
• Back off to embedding neighbors. Use the query embedding to find similar historical queries and pool their converted categories. They do exactly this on the SRL path (“brand names with high semantic similarity, ranked by embedding scores”), so the machinery exists.
• Back off to item-level signal. Retrieve items by text/embedding, then use those items’ categories as candidates. Works with zero query history.
• Let the LLM generate rather than re-rank when candidates are thin, with the guardrail catching invalid paths.

Note the structural asymmetry: the classification section is head-query-shaped, and the SRL section is where they explicitly solve the tail. The post presents both as “the LLM approach” without flagging that only one handles cold start.

“we use an LLM to re-rank them with injected Instacart context”

1. What context?
2. Do they use structured outputs? If not, how to ensure ranking output format is correct?

(a) For category re-ranking you’d want: candidate category paths with human-readable names rather than IDs; conversion counts or rates per candidate; a few exemplar item titles per category (this is what disambiguates a category whose name is vague); and the taxonomy’s own description of each node.

(b) In rough order of robustness:

1. Constrained decoding / grammar — vLLM guided decoding, Outlines, XGrammar. Invalid output is structurally impossible.
2. Provider-side JSON schema mode if using a hosted API.
3. Rank by index into the candidate list rather than emitting category strings. This makes hallucinating a nonexistent category impossible by construction and shrinks the output to a few tokens. For a re-ranking task specifically, this is the cleanest option and I’d expect it in a mature system.
4. Plain JSON prompt + parse + retry — weakest, but workable offline.

One clarification worth making: their guardrail (semantic similarity filter) is not a format check. It validates relevance, not schema. A malformed response fails before it ever reaches the guardrail.

Also: this is an offline/batch path, so retries are cheap and the format-safety pressure is far lower here than on the real-time SRL path.

“finally, we apply a post-processing guardrail. This filter computes a semantic similarity score between the embeddings of the original query and the LLM’s predicted category path, discarding any pair that falls below our relevance threshold.”

What model do they use for this score? Wouldn’t it struggle for the same reasons their original approach did?

[Search/Inferred] Two obvious in-house candidates: ITEMS (their Sentence-Transformers bi-encoder fine-tuned on conversion data) or the older MiniLM-L3-v2. ITEMS is the likelier one — already trained on query↔item pairs.

[Inferred] On the objection — the guardrail mostly escapes the original failure modes, for three reasons:

• Much easier task. The legacy classifier had to choose among thousands of categories. The guardrail only has to reject an obviously unrelated pair. A crude, low-resolution filter can be valuable where a crude classifier is not.
• Decorrelated errors. The LLM fails by hallucinating plausible-sounding but irrelevant categories; those are usually lexically and semantically far from the query, which is exactly what an embedding catches. The failure modes don’t overlap, which is what makes the two-stage design work.
• Tunable in the safe direction. Set the threshold loose so you only kill clear garbage. You’re not asking it to be right; you’re asking it to be right about egregious cases.

But the instinct is partly correct in one place: if the guardrail’s embedding model was trained on the same noisy conversion data, it inherits the same popularity bias — and may wrongly reject correct but rare query-category pairs. That would silently degrade exactly the tail queries this whole project exists to fix. The post doesn’t discuss this, and it’s the risk worth measuring before shipping.

“Our legacy system mined candidate rewrites from user session data”

What does this mean?

[Inferred] The standard technique is logging within-session query reformulation pairs. A user searches q₁ = “gluten free bread,” gets poor results, reformulates to q₂ = “gf bread,” and converts. You record (q₁ → q₂). Aggregate over millions of sessions, score by frequency and by the conversion lift of q₂ over q₁, then threshold.

Common variants:

• Full reformulation chains within a session, not just adjacent pairs
• Click/purchase-graph co-occurrence — two queries that lead to the same items are related
• Abandon-then-succeed pairs — highest signal, because the user has effectively labeled q₁ as insufficient

[Inferred] Why it caps near 50%: it can only produce a rewrite for a query that (a) someone has typed before and (b) was followed by a reformulation. Novel and tail queries have neither, by definition. It’s a memorization system with zero generalization — which is precisely the gap an LLM fills.

“This proved too ambiguous.”

How would they assess whether it was too ambiguous?

[Inferred] That reads like a qualitative, eyeballed finding, not a measurement. The rigorous versions:

• Type-distribution audit. Sample N queries, have raters label each generated rewrite’s type (synonym / substitute / broader / other) and usefulness. If output collapses onto synonyms — say 70% when you needed a spread — the instruction is under-specified. This directly diagnoses “ambiguous.”
• Incremental recall — the cheap quantitative version. How many items does the rewrite retrieve that the original didn’t? “one percent milk” adds roughly zero. No humans needed, and it operationalizes “not useful” exactly.
• Output-diversity check. Run the same prompt repeatedly at temperature; tight clustering means the model has silently picked one interpretation of an ambiguous instruction.

The generalizable lesson: “generate rewrites” doesn’t specify an objective. Splitting into three types is really specifying three separate objectives so each becomes individually evaluable — the evaluability is arguably the bigger win over the quality.

“This led us to design specialized prompts for three distinct rewrite types”

Are these three rewrite types all done every time for every query or is each of them triggered by its own unique conditions being met?

[Post] Not stated. It says each type is “handled by a dedicated prompt,” and that coverage rose above 95% “with 90%+ precision across all three types” — implying all three are generated broadly.

[Inferred] The likely design is generate all three offline, trigger conditionally at serve time:

• Generation: run all three across the precomputed query set. You can’t know in advance which you’ll need, and offline compute is cheap. “Coverage over 95%” is a claim about the generated inventory, not about firing rate.
• Serving: the post says rewrites matter “especially when the original query does not return sufficient results.” So the natural trigger is result-count or result-quality based. If the original returns enough good items, use nothing. If thin, escalate in order of intent-drift risk: synonyms first (safest, preserves intent) → broader (widens, some drift) → substitutes (changes the product, riskiest).

That escalation ladder is inference, not theirs — but it’s the only ordering that makes sense given the risk profile of each type.

“increased our query rewrite coverage to over 95%”

This feels like a very gameable metric in isolation?

Note they do pair it — [Post] “over 95% with 90%+ precision across all three types.”

[Inferred] Why coverage alone is nearly meaningless here: an LLM emits something for any input. Coverage measures the model’s willingness to answer, not quality. You can drive it to 100% by removing your abstain path. The legacy 50% was low because a mining system genuinely couldn’t produce candidates; an LLM’s pre-filter coverage is ~100%. So the honest framing is “50% from mining” vs “95% surviving guardrails” — and the interesting number is how much the guardrails removed, which they never report.

What would make it credible:

• Coverage on the tail specifically. Head coverage was never the problem, and an aggregate number is dominated by head volume.
• The coverage/precision curve as the guardrail threshold sweeps. A single point on that curve is unfalsifiable — you can always pick the flattering point.
• Incremental recall and conversion lift on queries where a rewrite actually fired.
• Null-result-rate reduction — the thing rewrites exist to fix, and the metric a business would actually pay for.

Coverage is an input metric. Nobody cares about it if it doesn’t move an output metric.

“90%+ precision across all three types”

How is this measured? Example?

Given the post’s own claim that QU lacks “direct feedback like clicks,” this is almost certainly human evaluation on a sampled set — possibly LLM-as-judge calibrated against human labels. The likely protocol, with the type-specific question a rater would answer:

• Synonym — “Would a shopper consider these the same product request?”
  “1% milk” → “one percent milk” ✓ · → “skim milk” ✗ (different product)
• Substitute — “If the original were unavailable, is this an acceptable alternative?”
  “Italian parsley” → “curly parsley” ✓ · → “cilantro” ✗ (different flavor profile)
• Broader — “Is this a strict generalization that still contains the original intent?”
  “2% reduced-fat chocolate milk” → “chocolate milk” ✓ · → “beverages” ✗ (too broad) · → “whole milk” ✗ (sibling, not parent)

Precision = rewrites judged correct ÷ rewrites emitted, computed per type. Recall isn’t reportable because there’s no enumerable ground-truth set of all valid rewrites — which is exactly why they fall back to coverage as a recall proxy, and why Q19’s critique lands.

[Inferred] Important caveat: precision defined this way measures semantic correctness, not usefulness. “one percent milk” would pass a synonym-precision check while being precisely the failure they described one paragraph earlier. The metric they report does not capture the problem they say they solved.

“we are now adopting context engineering to make rewrites more convertible, personalized, and session-aware”

What would those improvements look like in practice?

[Post] One concrete mechanism: “injecting user engagement signals, such as the top-converting product categories from their subsequent searches in the same session.”

[Inferred] Unpacking each:

• Convertible — bias rewrites toward products that actually sell and are actually in stock at the user’s selected retailer. Concretely: score candidate rewrites by the historical conversion rate of the items they retrieve, and gate on store-level availability. “Italian parsley → curly parsley” is worthless if that store stocks neither.
• Personalized — condition on purchase history and dietary pattern. “milk” from a user who has only ever bought oat milk should rewrite toward plant-based, not dairy. Same for brand affinity: a repeat buyer of one brand gets rewrites that preserve it.
• Session-aware — condition on queries already issued this session. Someone who searched “ground beef,” then “ricotta,” then “noodles” is making lasagna, so “noodles” should rewrite toward lasagna sheets, not ramen. This is the same idea as the post’s closing ambition of separating “lasagna ingredients” from “quick lasagna recipe.”

[Inferred] The cost, which the post doesn’t mention: all three push work from offline-and-cacheable to per-request, because the context is now user- and session-specific. That breaks the head-query cache — the exact mechanism that makes their economics work (Q22, Q29). That’s very likely why this is framed as future work rather than shipped.

“populating a cache for our most common “head” queries”

How to determine which queries? What threshold? Is it dynamic/time-windowed?

[Post] Never specified. Only: “the power-law nature of search traffic,” and — crucially, buried in the takeaways — “smart caching meant only 2% of queries needed real-time inference.”

[Inferred] That 2% is the answer in disguise: the cache is sized to cover 98% of query volume. The natural construction:

• Selection. Rank distinct queries by frequency over a rolling window; take the prefix covering 98% of impressions. Under a power law that’s a small fraction of distinct strings covering the overwhelming majority of traffic.
• Window. Should be rolling (30–90 days) and refreshed on a batch schedule, or it can’t track seasonality — “pumpkin spice” is head in October and tail in April.
• Dynamic promotion. Log cache misses; any query crossing a miss-count threshold gets queued for the offline teacher pipeline and promoted on the next run. This is what keeps the real-time model’s load pinned near 2% as traffic shifts.
• Key normalization. The cache key should be normalized text (lowercase, whitespace, punctuation, probably spell-corrected), or hit rate craters on trivial variants.
• Invalidation is the hard part they don’t mention: entries must expire when the taxonomy, catalog, or prompt changes, or the cache silently serves stale tags indefinitely. This is usually the biggest operational liability in a design like this.

What model used for Teacher and why?

[Post] Never named. Only “a powerful offline process,” “much larger frontier model,” “massive LLMs,” and Fig 5’s unlabeled “baseline.”

[Inferred] For work published November 2025, the plausible set is a GPT-4o/4.1-class, Claude Sonnet-class, or Gemini Pro-class hosted model, or a large open model (Llama-3-70B/405B) run in-house. Choosing Llama-3-8B for the student mildly suggests comfort with that family, but a hosted frontier API is the more common teacher because batch pricing makes it affordable.

Why a big model as teacher: you need the teacher’s quality to be the ceiling you distill toward. You only pay for it on head queries, in batch, with no latency constraint — so it’s expensive per call but amortized across a cached result serving millions of impressions. That amortization is the entire economic argument for the architecture.

Is their approach equivalent to standard teacher-student distillation or something different? Why this fine-tuning approach vs another?

It is not classical knowledge distillation, and the difference is worth being precise about:

• Classical KD (Hinton et al.) trains the student on the teacher’s full output distribution — soft logits, KL loss, temperature. Requires logit access and a shared tokenizer/vocabulary.
• What Instacart did is sequence-level distillation — take the teacher’s final generated text as a hard label, train with ordinary cross-entropy. Colloquially everyone calls this “distillation,” but structurally it’s supervised fine-tuning on machine-generated labels.

Why that variant:

• It works through an API. No logits needed, so the teacher can be a closed frontier model.
• It’s tokenizer- and architecture-agnostic. Teacher and student need nothing in common.
• The training code is just standard SFT. No custom loss, no distillation infrastructure.
• The decisive one: the teacher’s advantage comes largely from injected RAG context. Hard-label SFT is how you bake that retrieved context into the student’s weights — so the student doesn’t need to run retrieval at inference. That is what buys the latency. The student isn’t just a smaller model; it’s a model that has internalized a retrieval step.

One refinement worth naming: [Post] the teacher’s outputs pass “a post-processing guardrail [that] validates the tags against our catalog” before becoming training data. So it’s filtered / rejection-sampled distillation, which meaningfully raises label quality over raw teacher output.

Why LoRA over full fine-tuning [Inferred]: far less memory, hours instead of days, cheap to sweep variants, trivial rollback (keep one base, swap adapters). The cost is a lower capacity ceiling — which didn’t bind here because the task is narrow.

How did they know in the first place that the Teacher LLM works well for this task? Who evaluates the teacher?

[Inferred] The standard answer, and near-certainly what happened:

• A human-labeled golden set. A few hundred to a few thousand queries with expert-annotated SRL tags. Expensive — which is why it stays small — but you build it once and it anchors everything downstream. This is exactly the “costly and time-consuming human evaluation” the post cites in its challenges section.
• Stratified by traffic band (head/torso/tail) and query type (brand-led, attribute-heavy, ambiguous). A teacher that’s excellent on head and weak on tail is useless here, since the tail is the whole point.
• The catalog guardrail as an unsupervised check. Tags that fail catalog validation are wrong by construction — a cheap, large-scale lower bound on teacher error, no humans required.
• Downstream proxy. Do teacher-produced tags improve offline retrieval/ranking metrics? Noisier, but scales.
• The A/B test is the final arbiter, but it validates the system, not the teacher in isolation.

The structural problem worth sitting with: Fig 5 compares the student against the frontier model — i.e. the student is scored against the teacher, not against ground truth. If the teacher has a systematic blind spot, the student inherits it perfectly and the chart still shows 95.7% F1. “Matches the teacher” and “is correct” are different claims, and only a golden set separates them. They report the first and imply the second.

“For the real-time SRL model, we fine-tuned an open-source Llama-3–8B model using LoRA (Low-Rank Adaptation). The model was trained on the dataset from the offline “teacher” pipeline.”

What does this fine-tuning process look like concretely? How do I set up the necessary tools, datasets, infrastructure etc for this?

[Inferred] Concrete recipe as I’d run it today:

1. Data. JSONL of {messages: [system, user, assistant]} — user = the query (plus any context you want internalized), assistant = the JSON tag output. Target 5,000–50,000 examples. Hold out 5–10%, stratified by query frequency band so you can read tail performance separately. Dedupe aggressively; near-duplicate head queries will otherwise dominate the loss.

2. Framework, easiest → most controllable:

• Axolotl — YAML config, batteries included, best default choice
• Unsloth — fastest single-GPU (~2× speedup), ideal for 8B on one card
• HuggingFace TRL SFTTrainer + PEFT — most transparent, best when you need to customize
• LLaMA-Factory — good UI/CLI, broad model support

All wrap the same underlying PEFT LoRA implementation.

3. Hyperparameters, in order of impact:

• r = 16 (8–32 is plenty for narrow extraction); lora_alpha = 2r
• Target all attention and MLP projections (q, k, v, o, gate, up, down). Targeting only q,v is the old default and it underperforms — this is the single most common mistake.
• LR 1e-4 to 2e-4, cosine decay, 2–3 epochs, bf16
• Mask the loss to completion tokens only — don’t train on the prompt

4. Hardware. One 80GB A100/H100 fits 8B LoRA comfortably at bf16 with gradient checkpointing. On 24GB, use QLoRA (4-bit base via bitsandbytes) — quality is close for extraction tasks. Cloud: Lambda, RunPod, Modal, or SageMaker. A run at this scale is hours and tens of dollars, not a capital expense.

5. Eval, tracked per epoch. Three metrics: per-field F1 and exact-match on held-out, catalog-validation rate, and JSON parse rate. Parse rate should reach ~100% within one epoch — that itself confirms format learning is done and remaining error is semantic.

6. Merge and serve. peft’s merge_and_unload() → save as a plain HF model → serve with vLLM or TensorRT-LLM. Benchmark p50/p95/p99 at realistic concurrency, not batch-size-1 — that gap is where latency projects die.

7. Guarantee the schema in production. vLLM guided decoding (XGrammar backend) or Outlines, so parse failures become structurally impossible rather than merely rare.

“Merging the LoRA adapter weights directly into the base model”

What does this mean?

[Inferred] LoRA trains two small matrices per targeted weight — A (r×k) and B (d×r) — and inference computes h = Wx + BAx: the base projection plus a low-rank correction on a separate side path. Since BA has the same shape as W, you can precompute W' = W + BA once and store it. Inference becomes h = W'x — a single matmul, identical in cost to the unmodified base model.

Why it’s meaningfully faster:

• Eliminates two extra small matmuls per targeted module, per layer, per generated token. That’s roughly 7 modules × 32 layers × every token.
• Small matmuls are launch-overhead-bound, not FLOP-bound — so the cost is wildly disproportionate to the tiny parameter count. This is why 30% is plausible from a change that touches about half a percent of the weights.
• Removes the branching side path, which unblocks fused kernels and CUDA graph capture in the serving stack.

What you give up: you can no longer hot-swap adapters or serve many fine-tunes from one base-model copy in memory. A real cost in a multi-tenant setup; irrelevant here, since they serve exactly one.

Numerically it’s exact — merging changes nothing about model outputs beyond float rounding. It’s a free win when you don’t need adapter swapping.

What is typical acceptable latency in search systems like theirs? Given that they’re only focused on the first part of the pipeline, what latency budget are they allocated?

[Post] Their numbers: ~700ms out-of-the-box on A100 → 300ms target, achieved via adapter merging + H100. FP8 quantization would have cut another 10% (rejected for a recall drop). End-to-end impact: “only a marginal latency increase.”

[Search/Inferred] Industry context:

• E-commerce whole-page search typically budgets ~200–500ms p95 end-to-end; past ~1s, conversion loss is measurable. The canonical citations (Amazon’s “100ms costs 1% of sales,” Google’s 500ms/20%-traffic result) are old and over-quoted, but they’re why everyone budgets tightly.
• Within that, retrieval is usually tens of ms — [Search] Instacart’s own embeddings post cites under 8ms for on-the-fly query embedding — ranking is tens of ms, and the remainder is network, ads, and rendering.

[Inferred] The key structural point: 300ms for a single QU component is enormous — larger than many teams’ entire backend budget. It is only affordable because of the cache. At a 98% hit rate, the average added latency is about 0.02 × 300ms ≈ 6ms, which is why “marginal” is honest rather than spin. The 300ms is a tail cost paid by 2% of queries — and those are cold-start queries where the alternative was bad results anyway, so the user’s willingness to wait is higher and the regret from getting it wrong is larger.

That trade — accept a large latency hit on a small, high-regret slice — is the actual transferable design insight here, more so than the 300ms number itself.

Is “average scroll depth” just the number of items users scrolled past before clicking?

[Post] Defined only parenthetically: “(users find items faster).”

[Inferred] Roughly yes, but the exact definition changes the interpretation, and they don’t give it. The three common variants:

• Rank of first engagement — position of the first item clicked or added to cart. Lower is better. Most likely what’s meant, given “find items faster.”
• Max scroll depth — deepest item position rendered during the session, clicks or not. Measures effort, but conflates “found it fast” with “gave up fast.”
• Impressions before first add-to-cart — counts items seen rather than rank; handles grid layouts and lazy loading more cleanly.

Why the ambiguity matters: −6% is clearly good under the first reading. Under the second it’s ambiguous — abandonment also reduces scroll depth. A user who sees garbage and leaves immediately scrolls very little. Ruling that out requires pairing it with a conversion or engagement-rate metric. The complaints figure partially covers it, but not cleanly.

[Inferred] A grocery-specific wrinkle: users often scroll deliberately to browse alternatives even when results are excellent (“what yogurts do you carry”). That makes scroll depth a noisier relevance proxy in grocery than in, say, a support search box — worth knowing if you ever borrow the metric.

How would I convince a business leader that this project was worth doing?

Cost

• Reduced overhead from mode consolidation?

Benefit

• More conversions -> more revenue

How would you improve this system if you were building it today?

[Inferred] throughout, roughly ordered by expected value:

1. Build the golden eval set first. The biggest gap in the entire writeup (Q26). Without human-labeled ground truth, the teacher is unvalidated and the student is only ever measured against it. A few thousand stratified, expert-labeled queries is the highest-leverage artifact in the system — it makes every downstream decision cheap and every claim falsifiable.

2. Unify the three QU tasks into one model. Their own takeaway is “consolidate, don’t complicate,” yet they still ship three separate paths with different architectures. One fine-tuned model emitting categories + tags + rewrites in a single structured response is one forward pass instead of several, and it eliminates the cross-component inconsistency they named as the original problem (Q5). Multi-task training on a shared backbone usually helps each task, since they draw on the same underlying understanding.

3. Close the loop from serving back to training. Log cache misses, guardrail rejections, and null-result queries; feed them to the teacher on a schedule; retrain the student periodically. Prioritize queries where student and teacher disagree — that’s where the information density is. Their architecture is a one-way pipeline; making it a cycle is cheap and compounds.

4. Replace the exact-match cache with a semantic cache. String matching misses “gluten free bread” / “gluten-free bread” / “glutenfree bread.” Embed the query, ANN-lookup against cached keys, accept above a conservative threshold. They already run an ANN service. Higher hit rate translates directly into lower GPU spend.

5. Test whether you need 8B at all. A 1–3B model (Llama-3.2-3B, Qwen-3-4B class) may match on a task this narrow, at a fraction of the latency and cost. One experiment either saves real money or justifies the 8B — both outcomes are worth having.

6. Add abstention / confidence. The student always answers. Fig 5’s 96.4% precision means roughly 1 in 28 outputs is wrong with no way to identify which. A model that can say “unsure” lets you route those to a fallback — broader retrieval, or a synchronous call to a larger model with a longer budget — instead of confidently mis-tagging.

7. Audit the guardrail for popularity bias (Q15). Check whether the similarity filter disproportionately rejects correct-but-rare pairs. If so, make the threshold frequency-adaptive, or replace pure embedding similarity with catalog-grounded validation — “does any real item match this category path?” — which they already do on the SRL side.

8. Instrument the business metric. Conversion and null-result rate on the affected slice, in the A/B readout.

9. Modernize serving. Built today: server-side guided decoding, speculative decoding (SRL outputs are short and highly patterned, so draft-model acceptance rates should be excellent), and continuous batching in vLLM would likely land under 300ms without the H100 upgrade — which changes the hardware budget conversation entirely.

10. On their stated direction — session-aware, multi-intent — it’s right, but it breaks the cache (Q21). I’d resolve that by splitting rather than merging: keep query-level tags cached and stateless, and add a separate, small, cheap session-intent model whose output is combined at serve time. Putting session state inside the cached artifact destroys the 98% hit rate that makes the economics work.

“Attackers may launch attacks from compromised accounts making them even harder to identify from regular email”

What signals can you actually use if this happens? IP address, communication style, etc?

Per-sender behavior — writing-style deviation against that sender's own history, send-time-of-day shift, sudden send bursts, recipients never previously contacted

Mailbox configuration — new forwarding rules; rules that auto-delete or bury replies in Archive/RSS (the classic tell — the attacker hides the victim's incoming replies); delegate and permission changes.

Authentication / session — sign-in geography and ASN, impossible travel, new device fingerprint, legacy-protocol auth (IMAP/POP, which bypasses MFA), OAuth app consents

The IP address situation is a bit more complicated. It’ll be the same as the regular account holder (and thus not usable) if the attacker uses webmail or the mobile app. It’ll be different (and thus usable) if the attacker uses stolen credentials with an SMTP client which is much rarer.

“are identity of parties in the communication often targeted?”

What does this mean?

The intended reading: is this person's role one that attackers commonly go after? It's a prior on the identity, not a property of the message.

Role targeting — CEO/CFO (impersonated in BEC), AP clerks (invoice fraud), HR/payroll (W-2 and direct-deposit diversion), IT helpdesk (credential resets), executive assistants (authority by proxy)

“1 in 100,000 emails is advanced spear-phishing“

How was this estimate generated?

How a number like this necessarily gets produced: numerator = attacks their own engine caught, plus customer-reported and analyst-confirmed cases, plus retro-hunts. Denominator = total messages processed. So it's an observed detection rate, not true prevalence — a lower bound, because false negatives are by construction uncounted.

Three things that move it a lot:

• Denominator choice. Abnormal sits behind the existing gateway (API, post-delivery), so its denominator is already spam-filtered mail. That inflates the rate relative to raw internet email — the 65-in-100 spam is already gone.
• Definition. "Advanced spear-phishing" is their category, no external standard.
• Customer mix. Their customers are enterprises that bought advanced email security — plausibly more targeted than average.

“Account sign-ins, mail filters, and other account activity”

Do we typically have access to all of this information?

Yes, for the API-based model. Microsoft 365 exposes it through Graph API plus the Office 365 Management Activity API: Entra ID sign-in logs, mailbox audit logs, inbox-rule create/modify events, message content and headers, mailbox permissions and delegates, OAuth consents. Google Workspace has equivalents (Admin SDK Reports API for login/user-account audit, Gmail API).

Two caveats that matter:

• Licensing and consent. Full sign-in log retention depends on the customer's Entra ID tier, and you only get the scopes the admin consented to at install.
• Lag. Audit events surface with delay — historically minutes for the Management Activity API. So they are not available at the sub-second scoring moment for the message in hand. They feed the account-takeover detector and the feature store, not the inline decision. This connects directly to Q17.

“Malware in attachments”

How is this identified?

Four methods, cheapest to most expensive:

1. Hash / reputation — SHA-256 against known-bad corpora (VirusTotal, internal). Milliseconds; catches only knowns.
2. Static analysis — OLE/VBA macro extraction from Office files; PDF JavaScript, embedded files and launch actions; recursive archive unpacking; YARA rules; ML classifiers over PE headers and section entropy. Catches variants; defeated by packing.
3. Dynamic detonation — run the file in an instrumented sandbox (CAPE/Cuckoo lineage; commercially Joe Sandbox, VMRay) and watch process spawns, registry writes, network callbacks. Catches unknowns; takes 30 seconds to several minutes, and evasion-aware malware sleeps or checks for VM artifacts.
4. Multi-engine AV — several commercial engines in parallel.

“Adversarial attackers — To make matters worse, attackers actively manipulate the data to make it hard on ML models:”

For each of the methods listed here, what is one counter strategy each?

“The precision must be very high — to build a product to prevent email attacks we must avoid false positives and disruption of legitimate business communications”

Can we have different thresholds for different remediation severities (quarantine, block, etc)?

Yes, and this is standard practice: decouple score from action. The model emits a calibrated probability; a separate policy layer maps score bands to actions. Thresholds should be monotonically increasing in action severity, because each action has a different FP cost:

Three second-order points:

• Vary by attack class too. A suspected payroll-diversion BEC against the CFO justifies a lower bar than a generic promo-shaped phish, because the FN cost differs by orders of magnitude.
• Vary by recipient. Lower thresholds for finance, AP, and executives.
• The post-delivery architecture makes this much cheaper. Because Abnormal acts after delivery via API, it can decide late and revise — retracting a message when link detonation finishes or when the same campaign gets confirmed at another tenant. An inline gateway must decide once, immediately, with whatever it has.

One dependency: bands are only meaningful if the score is calibrated to an actual probability

“Is this pattern of communication unusual in any way?”

What are a couple of examples of stuff that would get flagged here?

Six kinds, organized by what's anomalous about the graph edge:

1. Edge never existed — first-time sender to this recipient, especially one that immediately asks for an action.
2. Edge exists, wrong direction or role — this vendor has always emailed AP, now it's emailing the CEO. Or the CEO mails a junior employee three orgs away they've never contacted.
3. Edge exists, properties changed — same display name, different sending domain than the previous 400 messages from that vendor. Or same person, new reply-to.
4. Volume/timing — an internal account suddenly mails 200 external recipients; a user sends at 3am local when they never have.
5. Missing expected structure — a "reply" that belongs to no thread the recipient has; a vendor invoice arriving off its normal monthly cadence.
6. Tenant-level shape — a domain that has never touched this organization now contacting five finance employees at once. That's campaign shape, invisible at the single-message level.

“Our ensemble of detection models”

Why ensemble as opposed to single model?

Five reasons, roughly in order of how much they matter here:

1. Adversarial robustness. A single model is one thing to reverse-engineer. The post says attackers A/B-test against filters — if they find the perturbation that flips your one model, they win everywhere. With detectors keyed on different data (text vs. graph vs. identity vs. links), evading all of them at once is much harder, and the evasion that beats one usually trips another.
2. Heterogeneous sub-problems. Invoice fraud, credential phishing and lateral ATO phishing have different base rates, signals and label volumes. A single model trained on all of it is dominated by the common class and underfits the rare, expensive one.
3. Precision-budget arithmetic. At a 1-in-a-million FP target, you can OR together several high-precision, low-recall detectors and accumulate recall without any one of them being perfect. A single model has to hit that operating point alone.
4. Operational independence. Ship and roll back separately, attribute failures to a component, patch a novel attack in hours with a signature while retraining runs for weeks.
5. Variance reduction. The textbook decorrelated-errors argument — the least important reason in this setting.

What do they mean by “redundant detectors”? Why do they use it and what does that look like in practice?

Redundancy ≠ ensembling. Ensembling combines weak signals into a better score. Redundancy means deliberately overlapping coverage: more than one independent detector capable of catching the same attack on its own. It's defense in depth for the classes where a miss is catastrophic.

Why: if detector A fails — new evasion, model drift, a feature-pipeline bug, a bad deploy — detector B still fires. The failure modes are meant to be uncorrelated.

For invoice fraud specifically, the stack would look like:

• The multi-modal deep model scoring the message end-to-end
• A dedicated invoice sub-model over attachment OCR and invoice layout
• A stateful check, not a model: does the bank account on this invoice match what this vendor has historically been paid to?
• Analyst heuristics — "reply-to ≠ from AND first-contact sender AND banking keywords AND urgency"
• Signatures on known campaign infrastructure

“for the convenience of including multiple modalities of data inside the same model”

A GBDT could also take in text and image embeddings as features though, right?

Yes — mechanically nothing stops you. The post's stated reason is convenience, and it undersells the real difference.

Four differences, and the first is the substantive one:

1. Gradient flow. Feed a GBDT frozen embeddings and the text encoder never learns from your detection labels. The figure shows the NLP sub-model's "Text Embedding" feeding a joint Neural Net — gradients propagate back into the encoder, so it learns representations tuned to this task (phishing register, urgency, payment language) rather than generic semantics. Trees cannot backprop into an encoder. This is the whole ballgame.
2. Dimensionality. A 768-dim embedding is 768 axis-aligned split candidates to a tree. Embedding information lives in directions, not coordinates, and trees split one coordinate at a time. GBDTs are weak exactly in the dense high-dimensional continuous regime. Note the converse: on genuinely tabular features GBDTs usually still beat NNs — which is why the figure retains a separate Tabular Sub-Model.
3. Variable-length data. An email has 0–N links, 0–N attachments, 0–N images. A tree needs a fixed-width row, so you hand-pool ("max link risk," "mean image score") and throw information away. A network can pool or attend in a learned way.
4. Engineering — the post's stated reason. One model, one training loop, one serving artifact, instead of a pipeline of separate embedders whose versions must be pinned and re-materialized every time they change. That last part is precisely the pain in his other Abnormal post on re-scoring past attacks.

Honest counterpoint: GBDT + frozen embeddings is a genuinely strong baseline, is what the post recommends starting with, and is cheaper to serve. The upgrade only earns its keep once you have enough positive labels to fine-tune — and labels are scarce here because of the base rate.

“we must be very careful that the model performs well autonomously and can handle distributional shifts”

How do they ensure this happens?

Pre-deployment

• Calibrate the score to a real probability (Platt/isotonic), correcting for negative subsampling
• Backtest on held-out future time slices, never random splits. Random splits leak badly here because campaigns repeat within a window.
• Re-score historical attacks with today's pipeline to confirm the new model still catches old ones.

At deployment

• Shadow mode: score live traffic, take no action, compare against production.
• Gradual rollout by tenant with a pre-agreed abort criterion.

Monitoring

• Watch the inputs, not just outputs: per-feature distribution drift (PSI/KL), null-rate spikes, cardinality changes. Feature-pipeline bugs look identical to genuine drift and are far more common.
• Watch firing rate per tenant at a fixed threshold — the earliest available warning.
• Asymmetry to plan around: precision is observable within days (customers report false positives); recall is not observable at all. You need proxies — analyst-labeled samples, purple-team and simulated attacks, cross-tenant confirmation.

Structural mitigations

• Per-tenant normalization. Features expressed relative to an org's own baseline transfer across orgs far better than absolute features.
• Keep the heuristic/signature layer, which can be updated in hours when the model degrades.

“Study the attacks, trends, and false negatives”

What are the sources for false negatives?

Seven channels, by how you learn about the miss:

1. End-user reports — the "report phishing" button in Outlook/Gmail. Highest volume, lowest quality.
2. Customer SOC escalation — the security team finds it in their own hunt and forwards it.
3. Incident-driven — post-breach investigation after money moved or credentials were used. Lowest volume, highest cost, most informative.
4. Retro-hunting on new intelligence — you learn a domain or hash is malicious later and sweep historical mail. Includes cross-tenant: an attack caught at customer B lets you re-search A and C. Abnormal's later material calls this "federated intelligence."
5. Delayed signal — the URL was benign at delivery and weaponized afterward (benign-then-swap). Only re-crawling links post-delivery surfaces these.
6. Threat-intel feeds and industry sharing — ISACs, vendor feeds, public reporting.
7. Phishing simulation vendors (KnowBe4, Proofpoint Security Awareness) — attacks in your stream with known ground truth.

The bias to name explicitly: every channel is biased toward attacks that were noticed. The FNs you never learn about are the successful ones never attributed to email at all. Measured recall is therefore optimistic by an unknown and unmeasurable amount — which is the deeper reason the post's base-rate figures in Q3 should be read as lower bounds.

“Build a portfolio of detection models, heuristics, and signatures”

What does signature mean here?

A signature is an exact-match rule keyed to a specific known artifact. It fires on identity, not on learned pattern — the narrowest, most brittle, most precise layer of the stack.

In email that means: a SHA-256 of a known-bad attachment; an exact URL, domain, or IP of known phishing infrastructure; a YARA rule matching a byte pattern inside a file; a specific sender address or subject string from an observed campaign; a structural fingerprint of a campaign's HTML template.

Why an ML-first company keeps signatures at all: response time and explainability. When a campaign is live right now, you push a signature in minutes; retraining takes weeks. And when a customer asks "why was this blocked?", a signature has a one-line answer.

“Ensure each sub-model is representing its sub-problem”

What are examples of sub-models representing sub-problems in this context?

All four feed a joint Neural Net → "Is Attack?"

[I] The design point that's easy to miss: the sub-problem is the modality, not the attack type. Short text is split from long text because a display name and an email body need different tokenization and different encoders — a subject line has no sentence structure to model, and a 24-character display name in a BERT-shaped encoder is mostly padding.

[S] The lesson in full: "Ensure each sub-model is representing its sub-problem well before trying to combine them into a larger network, or you will never make progress." Meaning: validate each branch in isolation — can the Link Sub-Model alone separate phishing URLs from benign ones? — before wiring it into the joint net. Otherwise a broken branch is invisible inside a joint loss, and you can't tell whether poor performance is an architecture problem or one garbage input.

[I] Sub-problems you'd expect beyond the figure: an OCR/vision model over attachment and landing-page images, an invoice-layout model, a per-sender writing-style model, and a communication-graph embedding.

“combine results of all models for a final decision on whether the email is malicious”

What would this look like in practice?

Four combination mechanisms, which are not mutually exclusive:

1. In-model fusion — the figure's approach. Sub-models feed a joint neural net; combination is learned, inside the network, and there's only one score at the end.
2. Stacking — each detector emits a score; a small meta-model (logistic regression or shallow GBDT) takes those scores plus context (tenant, attack class) and outputs the final probability. Needs clean held-out data or it leaks.
3. OR-of-thresholds — each high-precision detector has its own threshold; malicious if any fires. This is what "redundant detectors" implies, and it's the natural fit for recall-through-redundancy. Simple and explainable, but FP rates add.
4. Policy layer on top — score bands map to actions (Q7), plus overrides: allowlists, VIP-recipient escalation, tenant policy, and "if signature X fired, act regardless of score."

Realistically it's a hybrid: a learned score from the core multi-modal model, OR'd with a small set of high-precision specialist detectors and signatures, then a policy layer choosing the action.

One design constraint worth stating: attribution matters operationally. Customers ask why a message was actioned, and SOC analysts need to triage. So the combiner has to carry the firing reason forward — which argues against fully opaque fusion as the only layer.

“All this must be done at latencies of less than a second”

Where did this latency constraint come from?

Abnormal isn't inline. Their own architecture page describes a "pure API architecture that operates post-delivery," "outside the mail flow," with "no MX record changes," and notes that messages "briefly exist in the mailstore before remediation — typically measured in seconds."

[I] So the real constraint is time-to-remediate, not protocol timeout. The message is already sitting in the user's inbox. The window between delivery and retraction is the window in which someone can click, and mobile push notifications make that window very short — users read on phones within seconds of arrival. Every second is exposure.

The tension this creates, which the post doesn't acknowledge: several things on its own feature list cannot fit in a second —

• Sandbox detonation of an attachment: 30s to minutes (Q5)
• Headless rendering of a JS landing page behind a CAPTCHA: seconds
• Audit-log events (sign-ins, rule changes): minutes of lag (Q4)

So in practice this has to be a two-tier design: a fast path scoring on what's already available (text, headers, cached feature store, graph lookups) and a slow path that re-scores when enrichment lands — with the ability to retract a message after the fact. The post-delivery architecture is exactly what makes that second bite possible, and it's the same property that makes tiered thresholds cheap in Q7.

Read "less than a second" as an engineering target for the online scoring path, not an end-to-end SLA.

“How can we detect invoice fraud better? To do so we must deeply understand the natural language and images in an invoice to identify Abnormal patterns.”

What is invoice fraud exactly and why is it a challenge compared to the stuff they’re already doing?

What it is: getting a business to pay a real-looking invoice into an attacker-controlled bank account. Three variants:

• Vendor email compromise (VEC) — the attacker compromises the vendor's mailbox, sits inside a genuine payment thread, and sends a real invoice with changed banking details. This is what the post's "sit on a compromised account for months... inserting an illegitimate invoice into a conversation about payment at just the right time" bullet describes.
• Vendor impersonation — a lookalike domain of a real supplier; no compromise needed.
• Fake vendor — an invoice from a supplier that doesn't exist, aimed at a large AP department paying on autopilot.

Why it's harder than what they already do — seven reasons:

1. No malicious artifact. No malware, no credential-harvesting URL, no payload. Every signal a legacy gateway is built on is simply absent.
2. The text is legitimate business text. "Please find attached invoice #4417, net 30." Body-level NLP has nothing to grip — which is precisely why the post says you must understand the invoice, not the email.
3. The signal lives in the image/PDF. Detection requires OCR plus layout understanding: extract vendor name, remittance account/IBAN, amount, invoice number — a document-understanding problem, a different and much harder extraction task than reading an email body. That's what "images in an invoice" is pointing at.
4. It requires cross-message state. You cannot classify the message in isolation. You need a per-vendor payment-detail store: "this supplier has been paid to account ending 8812 for two years; this invoice says 4409." That's a data-engineering problem, not a model problem.
5. ~100× worse base rate. <1 in 10 million vs. 1 in 100,000. Far fewer positive labels, unchanged precision bar.
6. Every authenticity check passes. In VEC the mail genuinely comes from the vendor's real domain, valid SPF/DKIM/DMARC, inside a real thread with real history. The graph features that catch first-contact phishing all report "safe" — the exact opposite of the Q8 signals.
7. Highest per-incident cost, in both directions. Six- and seven-figure wire losses on a miss; blocking a real invoice stops the business paying its suppliers. Both FN and FP costs peak here simultaneously.

“How can we better identify account takeovers? This is a difficult anomaly detection problem”

Do they actually use traditional anomaly detection algorithms as either models or features?

I don’t think they use traditional anomaly detection algorithms.

• The dominant industry pattern is per-entity baselining and deviation scoring, not off-the-shelf unsupervised detectors. You model each user's own history and score how far today deviates — z-scores, quantiles, or a learned per-user profile. That is anomaly detection in the statistical sense, but it isn't "run an isolation forest on it."
• Why the classical algorithms underperform here: they find rare, and in enterprise email rare ≠ malicious. A salesperson traveling, a new laptop, a quarter-end send burst are all rare and all benign. At a one-in-a-million FP budget, an unsupervised detector's alert volume is unusable standalone.
• So the realistic role is as features into a supervised model, not as the decision-maker. Anomaly scores ("how unusual is this login for this user") become inputs alongside signals trained on labeled attacks.

How would you improve this system if you were building it today?

The attack side changed more than the defense side.

• LLM-written lures destroyed the fluency signal that 2019 NLP leaned on — no more grammar errors, and personalized spear-phishing now runs at spam economics.
• The attack went multi-channel: the callback step moved to cloned voice, SMS, Teams/Slack, and calendar invites. Email-only telemetry now sees a fragment.
• Targeting is automated: retrieval over public data to rank who has payment authority.

Seven changes, most to least important:

1. Model intent, not content. Since fluency no longer discriminates, stop asking "does this text look phishy" and model the request: what action, by whom, of whom, with what authority, at what point in a thread. "First-contact sender asks AP to change remittance details" is attack-shaped no matter how well written. This is an extraction-and-reasoning task an LLM does well and 2019 embeddings could not.
2. Replace the OCR→NLP pipeline with a vision-language model reading the rendered email, the attached invoice, and a screenshot of the landing page in one pass. That single component attacks all three of the post's stated open problems — invoice fraud, phishing sites, image-encoded text — at once.
3. Learned graph representations. 2019 used graph features; today use a GNN or temporal-graph model over the tenant's communication graph so relationship anomalies are learned rather than hand-engineered. Abnormal's 2026 material mentions graph neural networks, so this has happened.
4. Cross-channel identity fusion. One identity risk score across email, Slack/Teams, Zoom, calendar and the IdP — so a suspicious Teams message following a suspicious email is one incident, not two independent low-confidence events that each fall below threshold.
5. Adversarial evaluation as a first-class pipeline. Continuously generate LLM-based evasions of your own detector and train against them; treat detector robustness as a monitored metric, not a periodic red-team exercise.
6. Prompt-injection hardening. Any LLM component now reads attacker-controlled text. Email content must be data, never instructions — this is a new attack surface the 2019 design didn't have.
7. Fix some of it with workflow, not classifiers. For the highest-cost class, out-of-band verification of remittance-detail changes against a vendor record removes more residual risk than any marginal model improvement. Worth saying plainly rather than treating everything as a detection problem.

What I'd keep unchanged — these aged well: start simple and add complexity only when you can show it's better; heuristics as model features; redundant detectors; per-tenant baselining; calibration and thresholding discipline. Modest GBDT-on-tabular too — trees still beat NNs on genuinely tabular features.

It’s a little confusing to me that parents of items can be both brands (Coke) and sub-brands (Coke Zero). Why isn’t regular Coke a sub-brand of Coke the way Coke Zero and Cherry Coke are?

One way to think about this is that in their knowledge graph, a Sub-brand is created to capture a deviation from the core product (e.g., "Zero" for no sugar, "Cherry" for flavor). Because "Regular Coke" is the original standard, it has no deviating attributes to define it against the parent. It is the definition of the parent brand itself.

Depending on how the knowledge graph is used, having a Coke -> Regular Coke edge could increase latency without any actual benefit.

“Unstructured product description is passed to our in-house brand classifier”

What is their in-house brand classifier?

Exact model unclear from the blogpost but a reasonable assumption would be some sort of lighter-weight language model like BERT used as a closed-set classifier (trained on a limited set of labels).

“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction.”

Let’s say they used BERT as their initial classifier. Does lack of confidence here imply a relatively flat distribution (high entropy) over output probabilities?

That’s one way to measure it. We could also use margin (difference between top two predicted probabilities).

“The extraction output is passed to a second LLM, which retrieves similar brands and example item names from an internal knowledge graph to decide whether the extracted brand is a duplicate entity.”

Why do we need a second LLM for this? And how is it retrieving the similar brands and item names? Is it through embedding similarity search for the extraction output or by navigating the knowledge graph in some way?

By second LLM, I think they just mean the same model but called a second time with a different prompt.

1. The first call extracts the brand.
2. The second call, augmented through the similarity search, checks whether the extracted brand already exists.

I think the retrieval is just through embedding similarity based on the RAG system they describe later.

“the in-house classifier is retrained with the new annotations”

Does it make sense to retrain every time a new brand is added? And what would the retraining look like exactly?

I think retraining every time a new brand is added is probably not computationally efficient.

For the retraining step, if the classifier is a fixed-output model (e.g., a Softmax layer with N brands), the "head" of the model would need to be expanded to include a new node for the new brand.

For language models, you could freeze the earlier layers and simply retrain the final few layers of the model on the additional data + some older examples (to avoid catastrophic forgetting). LoRA could also work well.

If the cost of full retraining seems really low though, might as well do that.

“Last year, we stood up a model to label all organic grocery products”

Does it make sense to have a separate model to do this labeling as opposed to having it be just Organic/Not Organic be one part of the taxonomy mapping of a larger model?

One of the main downsides to having a single model do all sorts of taxonomy mapping is certain features may exist for some items but not for other items. Food items can be organic/not organic but batteries can’t. Having a model try to figure out what the feature value is for a feature that doesn’t exist for the item being considered can increase computational cost and negatively impact performance.

{Worth thinking through the other tradeoffs and how to make a final decision}

Where are the training data labels coming from in the organic product labeling use-case?

Since they mention “highest precision”, they must have some labels.

Unclear but either human labels or powerful LLM (that is really good at generating labels but too slow/expensive to run in production).

“missed cases where organic is misspelled / dropped or has a slightly different presentation in the data”

What do they mean by a ‘slightly different presentation in the data’? That it’s not included in the product title but might be present in an image or some other metadata?

Could also mean something like “Org," or "Org.” or the word is concatenated with other text e.g., OrganicMilk or Milk(Organic).

I feel like the LLM reasoning approach could have been used for the entirety of their catalog. Using a distilled language model for what I’m guessing is < 1 million unique items doesn’t seem like it should be that slow or costly, at least not anymore. Is that fair?

# of unique items isn’t the right number to think about; unique merchant SKU data is. Different merchants can encode their data differently and each of these different encodings would have to be processed.

Determining what would actually make the most sense in production would need more information about the cost and latency structure.

“LLMs conduct online searches of product information and pipe the search results to another LLM for reasoning”

What exactly do they mean by ‘LLMs conduct online searches’ and how is the first LLM here different from the second one?

It could look something like this:

1. The LLM is given a SKU (e.g., "Horizon Whole Milk 64oz").
2. The LLM generates a structured Search Query (e.g., site:horizonorganic.com "Whole Milk" ingredients organic certification).
3. A software wrapper executes this query via a Search API (like Google Search API or Bing Search API).
4. The raw HTML or snippets from the top results are scraped and fed back into the system.

The difference LLMs may not actually be different; they might just be referring to different calls with different prompts and context. It’s probably better to have a smaller LLM generate the search query and a larger one process the results to generate a response.

“Products from different categories are characterized by different sets of uniquely defining attributes. For example, an alcohol product is uniquely defined by attributes such as vintage, aging, and flavor.”

If we’re storing these in a database, does it make sense to have a column for each possible field or use a non-relational database?

Not a good idea to have a column for every field.

You can also use a hybrid structure where you have a column in a relational database storing category-specific attributes in a JSON.

We can also have a tall table - Entity-Attribute-Value (EAV) Model - where a single product could have multiple rows.

“For each unannotated SKU, we first leverage OpenAI embeddings and the approximate nearest neighbors technique to retrieve the most similar SKUs from our golden annotation set.”

What exactly are they feeding into the embedding model? Also, is this just equivalent to dynamic few-shot prompting?

An SKU at DoorDash is represented by a list of attributes, including:

• Item Name (e.g., "Dove Silk Glow Body Wash")
• Size/UOM (e.g., "500 ml")
• Brand (if available)
• Category

To generate the embedding, DoorDash likely feeds a concatenated string of these available attributes (e.g., "Dove Silk Glow Body Wash 500 ml") into the OpenAI embedding model. They are embedding the raw merchant-provided text to find semantically similar items in their human-verified "golden" dataset.

And yes, this process is functionally equivalent to Dynamic Few-Shot Prompting.

How do you pick the right number of examples to provide as context for the LLM in the previous step?

Test out different thresholds (for number and for similarity), and evaluate metrics of interest: accuracy, latency/cost etc.

“Ultimately, the generated annotations are used to fine-tune an LLM for more scalable inference”

What does this mean?

Could mean training a distilled model on outputs of a larger teacher model. Scalability is improved both in terms of the new model being faster/cheaper but also no API limits if we use an open-source model.

How would I convince a business leader that this project was worth doing?

Any cost-savings are definitely worth highlighting.

For other impacts, they’d need to figure out a way to connect improvements in the performance of these intermediate steps to actual downstream user impact (ads CTR, search fulfilment rate, etc).

How would you improve this system if you were building it today?

1. Use Contrastive Language-Image Pre-training (CLIP) to create unified embeddings that represent the text and the physical packaging in the same vector space.
2. Graph-Augmented Retrieval (GraphRAG)
   1. Instead of just finding "items that look like this one" via vector similarity, GraphRAG allows the LLM to traverse the Knowledge Graph's relationships (e.g., "This item belongs to the same distributor as Brand X")
3. Synthetic Data Generation
   1. Especially useful for edge-cases like misspellings.

“Attackers are often even using ML systems of their own to adapt their attacks!”

How would attackers use ML systems? Would they run their planned attacks through ML models to see if they would be picked up and adjust important features if so?

That’s one way to do it. They can either build their own models or use any of the following:

• Free/consumer filters — Gmail, Outlook.com — as a rough proxy.
• Secure email gateways bought as a legitimate tenant (a few hundred dollars a month), giving unlimited test sends.
• VirusTotal and public sandboxes, for the attachment/URL side.

There are other ways to use ML systems as well.

1. Reconnaissance and targeting — classifiers over scraped LinkedIn/corporate data to rank who has payment authority, cluster orgs by vendor relationships, and time the attack to real events (an acquisition, a quarter close).
2. Content generation — LLMs write fluent, on-register lures at scale, killing the grammar-error signal that older filters leaned on, and translate campaigns into new markets. Also style transfer: fed a few of a real executive's emails, mimic their cadence.
3. Identity spoofing beyond text — voice cloning for the callback step of a BEC, deepfake video for "verify me on Zoom." This is the fastest-growing branch since 2023.
4. Operational automation — ML to triage which compromised inboxes are worth working, and to run conversation threads with victims semi-autonomously.

“We cannot afford to throw out older attacks simply because they do not have the latest features. We must instead re-compute these features.”

Are there circumstances where it may be impossible to recompute those features?

Yes. Some examples include:

1. URL detonation doesn’t work because page is no longer live
2. Any internal features require underlying store to be append-only and timestamped
3. TTLs for data (because of GDPR erasure requests, data-residency rules, PII minimization commitments, contractual deletion)

“This includes changes to underlying detection code, new datasets, new features, and the development of new models.”

What are examples of new data sets that would affect the pipeline? Everything else makes sense to me, but I think I need an illustrative example of datasets to understand it better.

The model: a message arrives with its own contents. A dataset is a side table keyed on something in that message — sender domain, sender address, recipient, URL, attachment hash — looked up and joined in before features are computed.

Four sources of such tables:

1. The customer's own mail history. A per-tenant communication graph: for each sender-recipient pair, first-seen date and message count. Enables this vendor has emailed this AP clerk 400 times vs. never before. This class makes time travel hard — the correct join for a 2019 message is the graph as of 2019.
2. The customer's other systems. A new Okta or HR-directory integration gives you the real reporting structure, so "CFO wires request to a direct report" separates from "CFO asks someone three orgs away."
3. Third parties. A domain-intelligence feed keyed on sender domain, supplying registration date and hosting ASN — newly-registered-domain becomes computable the day you add it.
4. Internal analyst or model output. A curated list of known-good vendor domains, or campaign cluster IDs from another job.

“we make an unintentional code change that modifies a feature used by a model (but we do not retrain the model)”

What is a likely example of such a code change?

Four classes:

1. A refactor that alters a boundary case. Someone rewrites the sender-domain parser to handle subdomains properly. Correct by any reasonable spec — but a model that learned "this domain string is safe" now sees a different string on the same mail, and its lookup misses.
2. A dependency upgrade. Bump the tokenizer, language-detection library, or HTML parser. Text features shift slightly across every message; no line of your own code changed, so nothing flags it.
3. A change to a null or default. Missing sender-age gets -1 instead of 0, or a timeout that used to return "unknown" now returns "clean." The model learned the old sentinel's meaning and reads the new one as a real value.
4. A units or normalization change. Timestamps switch from local to UTC, a count becomes a rate, a score gets rescaled to 0–1. The feature is now more sensible and entirely wrong relative to the trained weights.

Why rescoring catches these and unit tests don't: the bug is a mismatch between two artifacts — code and model weights — that no single component's tests can see. Running the whole stack over Golden Labels surfaces it as a drop in rescored recall, which is precisely the argument the post is making for testing the stack rather than the parts.

“Process to evaluate the entire detection system to produce performance analytics like precision, recall. We call this “rescored recall.””

This isn’t equivalent to actual recall, right? Because they don't have all true positives.

Yes, it’s not equivalent.

“We must have correct and unbiased features”

The word "unbiased" implies the existence of biased features. Is this just a reference to time travel?

Yes, I think so.

“Should be able to update all historical samples”

Let's say the data you have doesn't let you update all historical samples, but only a subset. Do you just not use it then? That seems a little inefficient.

Not sure how Abnormal handles this internally but some alternatives to throwing it out include:

1. Mark the feature missing and let the model handle it. Tree models natively route missing values; for others you carry an explicit is_null indicator. Cost: the model learns a pattern from missingness itself, and if missingness correlates with age, it correlates with label. Mitigation is to inject the same missingness into recent samples at the observed rate so the indicator carries no time signal.
2. Restrict the metric's window, keep the sample. Report rescored recall on the full corpus for stack-wide features, and on the post-cutoff window for the new feature's contribution. The old attacks still train models and still count in every other evaluation. This is usually the right answer for a feature added six months ago.
3. Two-tier the corpus. A fully-hydrated recent tier for anything requiring complete features, plus a degraded historical tier used where partial features suffice. Explicit, but adds bookkeeping to every consumer.
4. Impute. Almost always the worst choice here. An imputed value on a rare attack is a fabricated signal on your most precious samples, and it's invisible downstream — unlike a null, which announces itself.

“Time travel ensures unbiased data and also helps us avoid future leakage: avoiding bringing any data from the future (to a past sample) that leaks labels into training features.”

What are a couple of examples of features that could be affected in this way?

1. Sender-recipient relationship count. A behavioral feature like "number of prior messages from this sender to this org." Recomputed today over the full history, an attacker domain that ran a 2019 campaign shows hundreds of messages — inflating it into a familiar sender. Worse, the count includes the campaign's own later messages, which exist because the attack happened. Time-traveled to the attack timestamp, the correct value is zero or near-zero, which is the actual signal.
2. Domain reputation from a threat feed. Query the feed today for a domain used in a 2019 attack and it returns "known malicious" — because the industry blacklisted it in response to that campaign. That feature is a near-perfect label proxy in training and worthless at inference, where the domain is hours old and unlisted. A model that leans on it will look excellent on Golden Labels and fail live.

“To do so, we rely on the most recent updated Golden Labels from the night before”

Is there a reason they're doing daily batching? Couldn't this miss recent attack patterns?

Why daily rather than continuous:

1. Cost. Full regeneration re-extracts features for every historical sample against time-correct join tables — a Spark job over years of data. Running it hourly multiplies compute by 24 for a corpus that changes by a fraction of a percent.
2. Labels aren't available in real time anyway. A message detected today may not be labeled for hours or days — customer reports, analyst review, retroactive discovery. A faster pipeline would mostly re-process unlabeled data.
3. Reproducibility. A dated, branch-tagged snapshot means two engineers comparing baseline vs. experiment are measuring against identical data. A continuously-updating corpus makes results non-comparable across runs.
4. Off-peak windows. Overnight batch avoids contending with production scoring for cluster capacity.

On your concern — the gap is real but it's not where the risk lives. Two things to separate:

• Detection latency — how fast a new attack pattern gets blocked. Not affected by this pipeline. That runs through the live scoring path and, presumably, faster mechanisms: signature/campaign clustering, analyst-pushed rules, threat-intel updates. Rescoring is an offline evaluation and training-data system.
• Evaluation and retraining latency — how fast a new pattern shows up in your metrics and your next model. Bounded at ~24 hours by this design, plus label delay, which likely dominates.

“We can either set up two configurations, “Baseline” and “Experiment,” or run this on two different code branches.”

What are the trade-offs between these two options?

What each can express

• Two configs, one branch. Only changes the config schema anticipated: swap a model, skip a stage, point at a different artifact. Anything requiring new code is out of reach.
• Two branches. Anything at all — new feature logic, changed extraction code, a new join, a restructured decision tree. This is the only option when the change is code, which is exactly the failure mode the post worries about in the "unintentional code change" paragraph.

Trade-offs

1. Confounding. Configs vary one declared knob against identical code — a clean A/B. Branches carry every difference between them, including drift from main if the branch is stale. Diagnosing "which of my six commits moved recall" is on you.
2. Compute. Configs can share upstream stages: run feature extraction once, fork only at MODEL_SCORING. Two branches generally can't share intermediates, since you can't prove the shared prefix is identical — so you pay roughly double.
3. Reproducibility. A config is a serializable object you can attach to a result and re-run later. A branch is a moving pointer; rebasing or force-pushing silently invalidates the comparison unless you record the commit SHA.
4. Path to production. A config swap is an experiment that then needs code written to ship it. A branch experiment is already the artifact you'd merge, so the thing you measured is the thing you deploy.

Rule of thumb: configs when the change fits the schema — cheaper, cleaner, more reproducible. Branches when it doesn't, accepting more compute and more attribution work. The post's stated aspiration, CI/CD auto-running rescoring on affected stages per code change, is the branch path automated.

“In many cases, GenAI-powered solutions are easier to maintain than traditional ML models”

Why is that?

1. The artifact you maintain is a prompt, not a training pipeline. A traditional supervised model for invoice parsing needs labeled data, a feature pipeline, a training job, a model registry, and a serving stack. Each is a thing that breaks. With GenAI the vendor owns the model; you own a text file and some post-processing.
2. Change cost is asymmetric. A new vendor with an odd invoice layout means, for a traditional model, collecting and labeling examples of that layout, retraining, revalidating, redeploying — days to weeks. For GenAI it means editing the prompt and rerunning evals — hours. This is the maintainability consequence of the flexibility bullet.
3. No feature engineering to keep alive. Document parsing models traditionally depend on OCR output, bounding-box geometry, layout heuristics, regexes per field. Those are brittle and accumulate special cases. An LLM absorbs that into the model's pretrained understanding.
4. No label drift treadmill. Traditional models degrade as the input distribution shifts and need periodic relabeling and retraining. GenAI doesn't require you to own that loop.
5. Fewer specialists needed. A prompt plus evals is legible to an engineer or analyst; a fine-tuned layout model needs someone who understands its training setup.

Why didn’t they ask the models making the predictions to generate confidence scores? Even if those scores are likely to be over-confident, that feels like something that can be addressed through calibration.

There are a few reasons why this might not work great (lack of independence, instability across runs, interpretability) but I think it’s still worth at least discussing.

““Calibrator” refers to using a separate single GenAI model to evaluate the outputs generated by other models and assign confidence scores”

1. Should this be a different model than those used for extraction? If so, which ones make sense for which use-case?
2. What would actually be included in the prompt for this model?

1. Should it be a different model?

Yes, for the same reason self-scoring fails: correlated errors. If the extractor and the judge are the same model, a misread "0" as "O" looks correct to both. A different model gives you an independent error distribution — the same property that makes their voting ensemble work.

Which model:

• Judging structured extraction against a document (their case) — needs strong document/vision grounding, since the judge must re-read the invoice image, not just inspect the text. That points to a frontier multimodal model, and specifically not the weakest member of the extraction ensemble.
• Judging free-text output with no ground-truth artifact — a cheaper text-only model is usually fine, since the task is coherence and consistency rather than verification.
• High-volume pre-filtering — a small fast model as a first pass, escalating only the uncertain cases.

2. What goes in the prompt?

Six things:

• The source document (image or OCR text) — without it the judge is grading plausibility, not correctness.
• The field definition, ideally the same one the extractor got, so both are answering the same question about "invoice total."
• The candidate value to be judged.
• A rubric that decomposes the judgment: is the value present in the document, is it in the right place, is the format right, is there a competing candidate elsewhere on the page.
• A required structured output — score plus a short justification, ideally with the justification first so the score follows from stated evidence rather than being emitted cold.
• A discretized scale with anchored labels (e.g. five bands, each defined) rather than a free 0–100. LLMs cluster at round numbers, and defined bands are more stable across runs — which was Spotify's stated complaint about calibrators.

“The calibrator model is also trainable by learning from feedback and can potentially improve over time.”

Does this mean fine-tuning, few-shot learning, prompt iteration or something else?

Not stated in the blogpost but my best guess would be few-shot examples plus prompt iteration, not fine-tuning.

The reasoning: their whole framing is that GenAI wins on fast development and low maintenance. Fine-tuning a judge would undercut both — it needs a labeled dataset of (document, extraction, correct-or-not) pairs, which is the labeling burden they were trying to escape by not training an extractor in the first place. And the invoice distribution keeps shifting as vendors change, so you'd be retraining continually.

“confidence scores generated by calibrators are difficult to interpret and sometimes counterintuitive”

In what ways is this true?

Difficult to interpret — the number has no defined referent. A voting score of 80% means four of five models agreed; you can state what generated it and reproduce it. A calibrator's 0.8 means whatever the model's next-token distribution landed on. It isn't a frequency, isn't tied to an observable event count, and can't be explained to an auditor as a control.

Counterintuitive — the specific ways this shows up:

• Non-monotonicity. Two extractions where one is clearly harder get similar or inverted scores. The judge is responding to surface features (clean formatting, plausible-looking numbers) rather than to actual difficulty.
• Clustering. Almost everything lands at 0.85–0.95, so the score doesn't separate the cases you care about. Then an occasional 0.3 appears for no visible reason.
• Score-justification mismatch. The judge writes a reason that identifies a real problem, then attaches a high score anyway — the text and the number are generated somewhat independently.

“the scores are not consistent across multiple runs”

What’s the underlying reason for this and are there any good ways to address it?

Underlying reason. Sampling is the obvious part — temperature above zero means the score token is drawn from a distribution, and if that distribution has mass spread across "8", "85", "9", you get different numbers on identical inputs. But the deeper issue is that the underlying distribution is flat. A well-grounded judgment concentrates probability on one answer and survives sampling; a weakly-grounded one doesn't. The instability is a symptom that the judge has no firm basis for the number, not just that decoding is stochastic.

Two secondary causes: LLMs are sensitive to irrelevant perturbations (ordering, whitespace, which examples are in the few-shot block), and even at temperature zero, batching and hardware nondeterminism in floating-point reduction make outputs non-reproducible run to run.

What actually helps:

• Discretize the output. Five anchored bands instead of 0–100. Sampling noise that would move 87 to 91 usually doesn't cross a band boundary. Biggest gain for the least effort.
• Sample it repeatedly and aggregate. Run the judge n times and take the median or modal band. This converts an unstable point estimate into a stable one — and note it's the same move as majority voting, which is where they ended up.
• Force the reasoning before the score, so the number is conditioned on stated evidence rather than emitted cold. Reduces variance but doesn't eliminate it.

“However, the methodology is not always transparent – it varies by provider, and for many models it is unknown how they were calculated.”

Don’t they all work the same way? If not, what are the differences?

The math is identical everywhere — log of the softmax over the vocabulary at one decoding step. What varies is everything between that number and what reaches you.

The three that actually matter for Spotify's method:

1. Availability. Not a uniform feature. Anthropic doesn't support logprobs at all; Gemini's support lives on Vertex AI rather than the standard API. A 2025 study of OpenRouter found only 164 of 813 endpoints (23%) returned them. A multi-vendor ensemble can't get logprobs from every member — which alone may explain why they dropped the approach.
2. Transformations applied before the number you see. Temperature scaling, top-k/top-p truncation with renormalization, repetition penalties, logit bias, and constrained JSON decoding all alter the logits first. Constrained decoding is the worst offender here: zeroing invalid tokens inflates confidence in whatever survives. Providers rarely document which of these apply.
3. Tokenization. Averaging over tokens makes the denominator tokenizer-dependent. "1,234.56" might be 3 tokens in one model and 8 in another, so the same extracted value gets a different average across ensemble members and isn't comparable.

Points 2 and 3 are what break their specific method — exponentiating an average of logprobs assumes a clean posterior, and under those transformations it isn't one.

“The confidence score was then calculated by exponentiating the average logprobs of the tokens.”

Why exponentiated?

To get back to a probability scale.

What does a single dot here represent?

Almost certainly a bucket of extractions grouped by confidence score, not an individual extraction. Two reasons:

• The y-values are accuracies — 60%, 75%, 96%, 22%, 12.5%. A single extraction is right or wrong, so accuracy is only definable over a group.
• The specific fractions give away small bucket sizes: 60% = 3/5, 75% = 3/4, 12.5% = 1/8, 66% = 2/3, 25% = 1/4. Roughly 8–12 dots per panel, each holding a handful of invoices.

“Majority voting is an ensemble method that selects the final output by choosing the most common response from multiple prompts or among multiple GenAI models.”

Are they using majority voting for the actual prediction too or just for the confidence scores? If it’s only for the confidence scores, couldn’t you potentially have the single extractor model have a different output than the majority vote?

I think they’re using it for both, which means the concern I raised is not applicable.

What models are generally considered best for this kind of use-case?

Frontier multimodal: Gemini 3 Pro led March 2026 invoice benchmarks (94.75%), with Azure Document Intelligence (90.52%) and Claude Sonnet 4.5 (90.27%) behind it. Claude Sonnet 3.5 degraded least as document quality dropped from clean PDFs to photos — resilience matters more than peak accuracy for real vendor mail.

Open-weight: Qwen3-VL-8B-Instruct is now state-of-the-art among open vision-language models.

Specialized engines (Azure, ABBYY, Rossum) remain competitive, particularly on tables and line items.

Two things that matter more than the rankings:

1. Pipeline structure beats model choice. A 2026 study on scanned financial documents found feeding full PDFs directly to a VLM was the worst configuration in every test; OCR-first multistage pipelines won by a wide margin. The document-specialized MiniCPM-o-2.6 also beat the much larger general-purpose Gemma-3-27b under identical settings.
2. For the ensemble, diversity beats individual accuracy. The voting signal comes from independent error distributions, so mix vendors and architectures. Logprob support is irrelevant here, so Anthropic's lack of it doesn't rule Claude out the way it would for the logprob approach.

“Literature often suggests using four to seven models in an optimal ensemble.”

What literature?

I haven’t found anything that supports this specific number so far so more extensive research may be needed.

In their use-case, they found that majority voting works best. Is that consistent with what the literature on this has found more generally?

Where the literature agrees. Consistency-based confidence is the mainstream approach, resting on Spotify's same premise: genuine knowledge produces reproducible answers, fabrications diverge. A recent head-to-head audit found self-consistency had the highest rank correlation with correctness on GPQA (ρ=0.31), ahead of verbalized confidence (0.21) and P(True) (0.19) — matching their ordering.

Their logprob failure also replicates, with a better diagnosis than the post offers. A 2026 paper on document field extraction notes these methods measure confidence in the generated token sequence, but extraction errors come from what the model can't observe — unreadable source material, ambiguous layouts, OCR noise. A model confidently transcribing OCR noise emits high logprobs for a wrong answer.

Where it's less flattering. That paper puts self-consistency across five calls at only 0.744 AUC on field extraction — better than logprobs, but not good, and at five times the cost. The audit concurs that no signal was a reliable standalone abstention score.

One way Spotify did better. Most of this literature tests self-consistency, resampling one model. They used cross-model voting, drawing on genuinely independent error distributions rather than one model's sampling noise — strictly stronger, and worth weighing against that 0.744.

The frontier is multi-signal. The 2026 paper argues for combining agreement with document-side evidence: image quality, spatial location, OCR text. That directly addresses the long-text-field problem they left unsolved.

“To calculate the final score, a weighted majority voting approach was implemented. Weight for each model was based on model accuracy and then normalized to sum to one.”

Do we need any kind of train-test split here?

Yes, and in two places.

1. The weights. "Model accuracy" has to be measured on labeled invoices. If you measure it on the same invoices you then report ensemble performance on, the weights are tuned to that set's noise — models that happened to get lucky on the hard cases get upweighted, and reported accuracy is optimistically biased. With only 5–6 weights the overfitting is mild, but it isn't zero, especially if the eval set is small (their Figure 1 buckets suggest it is).

2. Platt scaling — this is the bigger one. Platt scaling fits two parameters mapping raw confidence to calibrated probability, and it must be fit per field. Evaluate calibration on the fitting data and your ECE looks great by construction. Their Figure 3 shows before/after calibration curves; if those are in-sample, the improvement is partly guaranteed rather than measured.

What the setup should be: three splits, or nested cross-validation. One fold to estimate model accuracies for weights, a second to fit Platt parameters, a third untouched fold to report accuracy and calibration.

Why it matters more here than usual. The confidence score is a control — a threshold decides whether an invoice goes to a human. If calibration is optimistic in-sample, the real pass rate at a 95% threshold is worse than the number they'd show an auditor. Under SOX, an overstated control is the failure mode that actually costs you.

“we applied Platt scaling”

Why Platt scaling as opposed to isotonic regression?

Three reasons, in order of weight:

1. Sample size. Platt fits two parameters; isotonic fits a free monotone step function and needs roughly a thousand-plus labeled examples per field to avoid overfitting. They calibrate per field, and their plots suggest far less data than that.
2. Input granularity. With 5–6 models, majority voting yields only about six distinct confidence values. Isotonic would memorize an accuracy per value — effectively a lookup table with no ability to interpolate. A sigmoid smooths across levels.
3. Auditability. They chose linear weights over exponential explicitly for explainability. Two parameters is easier to document to an auditor than a nonparametric fit, and a sigmoid's smooth output avoids the step discontinuities that would make a threshold-based control jumpy.

Where isotonic would win: miscalibration that isn't sigmoid-shaped, such as good calibration mid-range but distortion only at the top end. Platt can't represent that — but at their sample sizes they likely couldn't detect it either.

“This approach requires careful tuning of distance thresholds.”

What would this tuning look like concretely?

Concretely, three separable pieces:

1. What you're tuning. A cosine-distance cutoff τ: two model responses join the same cluster if distance < τ. Too low and genuine paraphrases ("123 Main St." vs "123 Main Street") split into separate clusters, so agreement is understated and confidence collapses. Too high and real disagreements merge — the "0" vs "O" failure they hit — so confidence is overstated on a wrong answer. Related knobs, if you're being thorough: the embedding model, and the linkage rule (single-link chains transitively and will bridge distinct answers).

2. The objective. Not clustering quality. Build a labeled set of response groups where you know which responses are equivalent, then sweep τ and score against the two error types above. Because false merges are the dangerous direction in finance, weight precision over recall — e.g. maximize recall subject to false-merge rate under some ceiling — rather than optimizing F1.

3. Per-field fitting. One τ won't serve addresses, item descriptions, and vendor names — their embedding-distance distributions differ. That means a labeled set and a sweep per field, plus a held-out fold, since τ is a fitted parameter like the Platt coefficients.

Why it still failed for them: embedding distance is dominated by semantics, and "0" vs "O" is semantically invisible. No τ separates that case, which is why this is a threshold problem you can't fully solve by tuning the threshold.

“This approach requires extensive prompt engineering to ensure the selector prioritizes factual accuracy and consistency.”

How does factual accuracy fit into this? From my understanding, the selector model is just looking at the responses of the voter models, right? What facts are being verified?

Unclear from the post.

“we broke down long text fields into smaller, more manageable components (e.g., splitting addresses into street, city, state, and zip code)”

How did they do this break-down? Using a model or heuristic/regex?

Two possible readings, and the wording favors the first:

1. Split at the prompt, not after. Each model is asked for street, city, state, and zip as separate fields, so five separate votes happen on five short strings. Nothing needs parsing — the decomposition lives in the extraction schema. This fits "broke down long text fields into smaller components," fits their stated goal of improving agreement during voting, and fits their whole approach of solving problems by prompt adjustment rather than added machinery.
2. Split after extraction, parsing each model's full address string into components before voting. This is strictly worse for their purpose: if the parser is a regex it breaks on international addresses, which is their stated problem domain; if it's a model, you've added a component whose own errors now contaminate the vote.

Should we be concerned about the intra-model correlation in the prompt permutation phase? A 94% score suggests a high degree of confidence but if the same models are being used, I feel like it’s somewhat exaggerated.

I think this is a reasonable concern but maybe mitigated somewhat by the calibration stage.

How would I convince a business leader that this project was worth doing?

Hard to say without more insight into the use-cases of human-in-the-loop decisions and meeting regulatory requirements.

How would you improve this system if you were building it today?

1. Add document-side signals to the score

The biggest weakness is that every signal they use is model-side. Agreement can't detect a failure all models share, and their unsolved "0" vs "O" case is exactly that. Current work on extraction confidence combines vote agreement with evidence from the document itself: image quality metrics, whether the extracted value is spatially locatable on the page, agreement with an independent OCR pass, and agreement between two structurally different extraction calls. A wrong character is invisible to voting but visible to a string comparison against OCR text.

2. Use business validation as a free, hard signal

Invoices are internally redundant in ways they never exploit: line items should sum to subtotal, subtotal plus tax should equal total, currency should match the header, the invoice number should match the PO's, the vendor should exist in the master file. These are deterministic checks with no false-confidence failure mode. A total that fails arithmetic should be low-confidence regardless of unanimous agreement, and a total that passes deserves a lift. This is cheaper than any additional model run.

3. Restructure the ensemble spend

Instead of running 5–6 frontier models on every invoice: run 2–3 on the first pass, and escalate to the full ensemble plus permutation only when the cheap signals disagree or validation fails. Most invoices are native digital PDFs where accuracy is 98–99%; the cost belongs on the photographed and multilingual tail.

Two related fixes: two-stage voting for the permutation phase (per-model modal answer, then vote across models) so correlated prompts don't inflate the denominator; and mixing architectures deliberately — a specialized engine like Azure alongside frontier VLMs — since diversity is what the vote is actually measuring.

4. Fix the pipeline, not just the aggregation

A 2026 study on scanned financial documents found feeding full PDFs directly to a VLM was the worst configuration in every test; OCR-first multistage pipelines won by a wide margin. If they're still passing whole documents, upstream structure will buy more than another voter.

5. Tighten the evaluation

The post shows no train/test discipline for the weights or the Platt parameters, and the calibration curves may be in-sample. Nested splits, reported AUC and AURC rather than just correlation, and a stated false-merge rate — because under SOX an overstated control is the failure that costs you.

Does the model run on all items the user adds to their cart or just the ones that turn out to be unavailable?

For all ordered items, not just the ones that turn out to be unavailable. Serving is almost certainly a precomputed item→candidate-substitutes table refreshed in batch, not a live forward pass over the whole catalog per cart.

“then proceeded to binary classification, and eventually pursued a deep learning recommendation model”

The way this is phrased makes it seem like the deep learning model isn't doing binary classification but something else instead?

The phrasing is imprecise - the DLRM is still doing binary classification.

Why did they use TF-IDF cosine similarity as opposed to semantic embeddings (like BERT) at the start?

No reasons presented in the blogpost but possible reasons include:

1. Off-the-shelf BERT is a poor fit for product titles specifically. Grocery SKU names are short, keyword-dense strings with brand tokens, sizes, and units — "Coca-Cola Classic 12 fl oz Can 12pk." Generic BERT is pretrained on prose and, without fine-tuning, its sentence embeddings underperform even simple lexical baselines on this kind of text; it also tends to wash out the exact-token matches (brand, pack size) that carry most of the signal here. TF-IDF preserves those.
2. Operational cost. TF-IDF is a sparse matrix and a dot product — cheap to compute over a large catalog, trivial to refresh when SKUs change, and fully inspectable when a recommendation looks wrong. A transformer at phase 1 means GPU inference, an embedding store, and a debugging surface you can't eyeball. For an MVP whose main job was to unblock feedback collection, that overhead buys little.

Why did they only use the item’s name as opposed to descriptions to find similar items?

No reasons presented in the blogpost but possible reasons include:

1. Descriptions largely didn't exist. DoorDash's grocery catalog was assembled from merchant feeds and in-store data collection, not authored by DoorDash. Names are the one field every SKU has; descriptions are missing, truncated, or inconsistent across merchants for a large share of items. A feature you can only compute for part of the catalog directly caps the coverage metric they later tracked.
2. For groceries, the name carries nearly all the discriminative signal. "Bush's Best Cut Green Beans 14.5 oz Can" already encodes brand, product type, form, and size — the exact axes that determine a good substitute. This is unusual: for restaurants or apparel, descriptions matter much more. Grocery SKU names are effectively structured records rendered as strings.
3. Descriptions would have actively hurt a TF-IDF model. They're mostly marketing boilerplate and shared brand copy. Under bag-of-words cosine similarity, two unrelated products from the same manufacturer share long stretches of identical text and score as similar. IDF down-weights terms common across the whole catalog, but not terms common within a brand or category — so the noise survives. Longer documents also dilute the weight of the few tokens that matter (the pack size, the brand).

“our catalog team built out a well-defined taxonomy that let us apply heuristics on top of the text-based similarity score to restrict recommendations to relevant categories”

I didn't fully understand this. What is an illustrative example?

What it means: two filters in sequence. TF-IDF says "these names share words." The taxonomy says "these are the same kind of thing." Rank by similarity within a category node; drop anything outside it regardless of score.

Example. Customer orders Coca-Cola 12-pack cans. TF-IDF alone surfaces Pepsi 12-pack and Diet Coke 12-pack (good), but also Coca-Cola Gummy Candy, Coca-Cola BBQ Sauce, and a Coca-Cola T-Shirt — all scoring high on the brand token. The taxonomy cuts the last three: wrong category node.

“chose to use LightGBM for this phase because of both its relatively high performance with minimal hyperparameter tuning”

High performance as measured on their training data or based on prior literature?

Unclear from the blogpost. It’s worth noting that "performance" here could mean predictive accuracy or computational efficiency. LightGBM's actual differentiator versus XGBoost is speed — histogram-based splitting and leaf-wise tree growth — with accuracy roughly comparable

“this model combines principles from approaches based on collaborative filtering and predictive analytics”

In what ways does this model combine those principles?

Collaborative filtering side = the embeddings plus the explicit interaction step. Learning a vector per item and scoring pairs by dot product is matrix factorization, the canonical CF method. DLRM computes pairwise dot products among all embeddings, so that layer is essentially a bank of MF models. The signal is behavioral: which items co-occur or get approved together, with no notion of what the item is.

Predictive analytics side = the two MLPs over dense features. Continuous features (price, size, popularity, similarity scores) go through the bottom MLP and, after interaction, the top MLP — a standard supervised classifier on engineered features, which is exactly what phase 2's LightGBM was. The signal is attribute-based.

Why combining matters here: pure CF fails on cold-start items with little feedback; pure feature-based models miss preference patterns not encoded in attributes (the Pepsi 12-pack over 2-liter Coke case). DLRM lets both flow into the same sigmoid.

One caveat: DoorDash's item embeddings were pretrained on search behavior, not learned fresh from substitution labels. So the CF component partly comes in as a frozen input rather than being fit end-to-end — the post doesn't say whether those embeddings were fine-tuned.

“categorical features (or in this context, items in our catalog) are processed as embeddings”

How are items represented in this model? A single embedding per item or broken down into different features (some of which would be categorical features represented as embeddings)?

Most likely a single vector per item, not a decomposition into brand/size/category sub-embeddings. The post's own phrasing points this way, and the semantic embeddings it inherited are whole-item vectors by construction. Attribute-level features are named as future work ("organic," "kosher," image embeddings), which implies they weren't decomposed at the time. Standard DLRM would allow additional categorical fields — taxonomy node, brand, merchant/store, each with its own table. Plausible here given the catalog taxonomy already existed from phase 1 but bot mentioned either way.

Note that we have at least two item embeddings per scoring pass. The task is pairwise — score candidate j as a substitute for ordered item i. So both IDs get looked up, and the explicit interaction layer's dot product between them is where most of the signal lives. The post never says this, but the task formulation requires it.

“Next, feature interactions are computed explicitly”

What does this mean?

In DLRM, "explicit interaction" is a specific operation: take every embedding vector plus the bottom MLP's output, and compute the dot product between every pair. Those scalars get concatenated with the dense vector and passed to the top MLP.

"Explicit" contrasts with "implicit." If you just concatenate all inputs and feed an MLP, the network can in principle learn multiplicative relationships between features, but it has to discover them through stacked additive layers — slow and sample-hungry. DLRM instead hard-codes second-order terms, borrowing from factorization machines. Here that's very natural: the dot product between the ordered item's embedding and the candidate's embedding is a similarity score, which is the core signal for substitution.

What is the difference between the bottom MLP and the top MLP?

Why the bottom MLP exists at all (inferred): dimensional compatibility. The interaction layer takes dot products between vectors, so every participant must have the same width. Item embeddings might be 64-d; the raw dense features are some arbitrary count of scalars (price, size, popularity, TF-IDF similarity). The bottom MLP projects them to 64-d so the dense side can participate in the same pairwise dot products as the items. Without it, dense features couldn't interact with embeddings at all — you'd be stuck concatenating them at the end.

Why the top MLP exists: the interaction layer outputs a bag of scalars with no ordering or weighting. Something has to learn which of those pairwise agreements matter and how they combine nonlinearly. That's the top MLP — it's the actual classifier, and it's where higher-order effects beyond second order get modeled.

Rough intuition: bottom MLP is a translator getting dense features into embedding space. Top MLP is the judge deciding the verdict from all the evidence.

“the embeddings are trained on the search behaviors of DoorDash users”

In 2-3 sentences, explain how the search behavior is used to train the embeddings.

Search logs supply pairs of things users treated as equivalent — a query and the item they clicked, or two items clicked under the same query. A twin (Siamese) network encodes each side with shared weights and is trained with a contrastive objective: pull matched pairs together in vector space, push randomly sampled non-matches apart. The result is that items people search for interchangeably end up nearby, which is why green beans land closer to peas than to corn.

“For example, as shown in Figure 4, the LightGBM model recommended canned corn as a substitute for canned green beans. The deep learning model, however, recommended canned green peas, because item embeddings accurately represent that beans are more similar to peas than corn.”

Is there something intrinsic to these models that leads to LightGBM performing worse for this example or is it more random?

It's intrinsic — but the cause is representation, not "trees are worse than neural nets." Two distinct mechanisms:

1. Nothing in LightGBM's feature set separates corn from peas.
Trees split on features you hand them. Under plausible phase-2 features — name token overlap, taxonomy node, price, size, popularity, per-pair feedback rate — canned corn and canned green peas are near-identical for a green-beans query. Same node (canned vegetables), same price band, same "canned" token. With no discriminating feature, the model falls back on the strongest remaining signal: popularity. Canned corn outsells canned peas, so it wins. That's a systematic bias, not noise.

2. Even given embeddings, a tree can't cheaply express pairwise similarity.
Feed LightGBM the 64-d item vectors as columns and it still can't compute a dot product — axis-aligned splits approximate a similarity surface only with an enormous number of splits. DLRM's explicit interaction layer does it in one operation. This is a genuine architectural gap for pairwise tasks.

“We then compared what percentage of the algorithm’s recommendations were matched by human-curated substitutions.”

Is this equivalent to precision, recall, or something else?

Precision@k

“coverage (percent of ordered items with recommendations)”

Shouldn’t this be 100% by construction?

No — and the reason is that the model was deliberately allowed to abstain.

A pairwise scorer over the whole catalog will always return something if you take the argmax. But shipping the argmax regardless of score is bad: a wrong substitution is worse for the customer than no suggestion, since the fallback (chat with the Dasher, or a refund) already exists and is acceptable. So there's a confidence threshold, and items whose best candidate falls below it get nothing.

“We also plan to develop personalized recommendations because we’ve observed that customers have highly individualized substitution preferences”

What would be the best way to incorporate personalization into this system?

Three places personalization can enter, by pipeline stage:

1. Input layer — a user ID embedding as an additional categorical field.
DLRM-native: add a user embedding table, and its dot products with the ordered-item and candidate-item embeddings fall out of the existing interaction layer for free. Architecturally the cleanest. Directly hits the sparsity wall above, so it only works for the head of the user distribution.

2. Feature layer — user-level dense features into the bottom MLP.
Engineered signals that generalize across users rather than identifying them: this customer's historical brand-switch rate, private-label acceptance rate, size-flexibility rate, organic-purchase share, price sensitivity, past thumbs-down rate by category. Each is estimable from a handful of events and degrades gracefully to a population prior when data is thin.

3. Post-scoring layer — constraints and re-ranking on a global model.
Hard filters from stated preferences (dietary restrictions, allergen avoidance, no-substitute flags) plus boosts from the customer's own purchase history — a candidate they've bought before should outrank an equally-scored one they haven't.

How would I convince a business leader that this project was worth doing?

1. Come up with some estimate of the cost to DoorDash of
   1. Users not receiving the item they wanted because out of stock
   2. Friction from manual substitution
2. Calculate how much of that cost has been mitigated through the algorithm

If you’re A/B testing, you can also look at various user engagement and revenue metrics

How would you improve this system if you were building it today?

Ranked by expected value.

1. Condition on real-time store-level availability.
The system as described scores substitutes for an item in the abstract. But a recommendation is worthless if the suggested item is also out of stock in that store, and the Dasher is standing in the aisle with a list. I'd make the candidate set store-and-time-specific: score P(good substitute) × P(in stock at this store now), where the second term comes from an availability model fed by Dasher found/not-found reports, POS feeds where available, and recency of last confirmed pick. This is the largest gap in the original design and it's orthogonal to model quality — a mediocre recommender that only suggests things actually on the shelf beats an excellent one that doesn't.

2. Generate the missing metadata with a VLM instead of waiting for merchants.
The post's own next steps name richer attributes (organic, kosher) and image embeddings as future work, and my earlier answer on why they used names only came down to descriptions being absent. That constraint is largely gone. Run a vision-language model over product images and packaging text to extract a structured attribute record per SKU: brand, form (canned/frozen/fresh/dried), net quantity and unit, flavor, dietary flags, allergens, private-label indicator. Those become explicit features rather than something the model has to infer from a name string, and they make the failure modes debuggable — you can point at why a swap was blocked.

3. Fix the label.
Thumbs-up/down on a shown recommendation is a preference proxy collected before the fact, and it's only observed for pairs the system chose to display — classic selection bias, where the model is trained on its own past decisions. I'd train on realized outcomes instead: was the substitute accepted, kept versus refunded or returned, and how was the order rated. Combine with logging propensities so offline evaluation can use inverse-propensity or doubly-robust estimators rather than accuracy on a biased sample. This is unglamorous and probably worth more than any architecture change.

4. Split retrieval from ranking.
A single pairwise classifier over the catalog forces a precomputed table. I'd use a two-tower encoder plus approximate nearest neighbor search for candidate generation (cheap, refreshes as the catalog changes, handles new SKUs), then a cross-encoder reranker over the top ~100 that can see the ordered item, the candidate, store context, and the customer together. Retrieval gets coverage; reranking gets precision. It also makes the personalization layer from the last turn much cheaper to apply, since it only touches the shortlist.

5. Hard constraint layer, separate from the model.
Allergens, dietary and religious restrictions, alcohol, pharmacy, infant formula, and large price deltas should be enforced by rules that sit outside the scorer and cannot be overridden by a high probability. The errors that damage trust most are categorical, not marginal — and a learned model trained on aggregate approval will occasionally produce them.

6. Evaluate on the coverage/approval frontier, not a point.
The precision-against-golden-set metric from phase 1 is rank-insensitive and head-biased. I'd report the full curve of approval rate versus coverage as the abstention threshold sweeps, broken out by category and by customer history depth. That prevents shipping a model that looks better only because it helps fewer people.

“We compared the Abnormal detection system—a multi-stage pipeline combining behavioral models, downstream rules, and an LLM critic”

How does an LLM critic fit into the Abnormal pipeline? What is its role?

Abnormal says their classification cost is $20/million messages. Based on frontier model cost, LLMs only fire for 0.1% of messages.

This suggests that the critic sits off the default path, invoked on a small minority of messages after cheaper stages have already decided.

The use of the word “critic” could mean that (assuming the prior stages are tuned more for recall than precision) the LLM is being used to suppress false positives but this isn’t stated in the blog post.

A full pipeline could look like

1. Lighter-weight model/model cascade with a low threshold sends all cases above a certain score to the next step
2. LLM reviews likely positives based on specified guidelines

“the efficacy comparison reflects this specific evaluation set, not production performance.”

Are there any obvious gaps in the curation of the data set? Beyond the fact that Abnormal’s models are trained to be aligned with Abnormal’s security analyst decisions

1. Selection — the attack set is conditioned on Abnormal having caught it. Attacks Abnormal never flagged don't reach that queue.
2. Selection — the safe set is likely hard-negative enriched.
3. Label taxonomy — "safe" and "graymail" overlap by construction
4. Temporal — contamination is checked in one direction only
5. What each system was shown

“returns a decision—attack, spam, graymail, or safe”

How does Abnormal typically deal with these four categories of decisions? What is the action taken based on the decision?