SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries

Abstract

Long-running LLM agents act through tools. A single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark protocol for measuring that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, security, and other domains.

Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls. The labels split nearly evenly between proceed and hold, so the two error directions get near-identical numbers of chances. The protocol, not the count, defines the benchmark: a model sees the proposed action and available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly.

Across 30 model conditions, the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The direction holds under sensitivity checks: excluding the 17 rows where the runner emitted a derived risk flag gives 19.4% and 1.25%, and relabeling against an unadjudicated three-rater majority gives 41.1% and 0.36%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). A model can win the majority vote while flipping between answers across trials. General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary. More reasoning can repair a weak gate while leaving a calibrated gate flat or slightly worse. The same grid selects open-weight repair targets for post-training at fixed-or-lower under-refusal. The public leaderboard is at steerbench.com.

Introduction

The action-boundary commit is the moment the human in the loop will not see if everything works. After commit, the consequence is in the world and is no longer steerable. Public incidents from the LLM era and from earlier automated-decision systems document the failure pattern: agents and automated decision systems that judge the boundary incorrectly produce tribunal rulings, regulator settlements, class actions, and operational harm. The pattern is reproducible, and a labeled benchmark of those decisions is the natural measurement tool.

Frontier labs already ship action-boundary reviewers. In Codex Auto-review, OpenAI routes a planned action and recent context to a reviewer before execution; in Claude Code Auto Mode, Anthropic uses an action classifier to reduce approval fatigue while blocking risky tool calls. These are product and engineering references, not open benchmark papers. They show that the boundary is operationally real.

The benchmark gap is narrower and sharper: the field lacks an open, incident-anchored, bidirectional benchmark with open training views for workplace action-boundary gates. XSTest, OR-Bench, SORRY-Bench, and HarmBench measure chat refusal where the question is whether the model produces a particular kind of text reply. SWE-bench and GAIA measure final task completion where the question is whether the patch passes tests or the answer matches ground truth. HiL-Bench, ClarifyBench, and AskBench measure ask-or-act when the agent lacks information. ST-WebAgentBench scores web-agent policy compliance with a metric that falls only when an agent acts beyond its permissions, never when it acts short of them. Independent permission-gate work such as AmPermBench is adjacent, and reports over-blocking where the authorization is deliberately ambiguous. AgentAbstain, concurrent work, tests act-or-abstain decisions in executable sandboxes under ambiguity, tool failure, and runtime discovery. SteerBench-Work focuses on public incidents, paired incident mirrors, both under-refusal and over-refusal, risk-resolved workplace commits, and open training views.

This paper contributes:

1. A corpus that scores the live steering decision. The current release places a model at the commit moment with the user request, the proposed action, and the evidence on hand. The scored field is the binary proceed-or-hold decision. Scenarios are anchored to public incidents so each label is auditable. We have not found a prior benchmark that combines public-incident anchoring, paired mirrors, and pre-commit labels for both unsafe action and unnecessary interruption at workplace action boundaries.

2. Incident-mirror methodology. A 13-scenario mirror subset preserves the surface shape of public incidents while inverting the verification state, testing whether the gate follows the changed evidence or keeps giving the answer the incident is associated with.

3. Both steering errors in one suite. The release scores under-refusal and over-refusal from the same scenarios and the same scored field, so a model cannot lower one by silently trading it against the other. Bidirectional scoring is not itself new: HiL-Bench's Ask-F1 balances over-asking against silent guessing when the agent lacks information. What is scored here is both directions at a boundary where the information is sufficient, and the decision turns on authority, evidence resolution, and irreversibility.

4. Irreversibility-weighted severity reporting. A secondary severity metric beside mean trial accuracy, modal-of-5, and pass5, showing the operational severity of remaining misses without changing the primary unweighted leaderboard rank.

5. Domain and action-effect stratification. Every scenario in the release carries workplace domain and action-effect metadata. The run reports stratified results so a model's weak areas are visible rather than hidden in one global score.

6. Training and evaluation views in the tinker-cookbook shapes. The same scenario schema carries the labels a training run needs. In the planned 500-scenario corpus, examples are split by scenario family into train, development, and sealed test sets, so final lift is measured on scenarios not used for supervision or recipe tuning.

Related work

Product action gates. OpenAI's Codex Auto-review and Anthropic's Claude Code Auto Mode are production examples of action-boundary review. We cite them as engineering references: they motivate the gate as a real product boundary, but they do not release an open incident-anchored benchmark or open training recipe for the gate. AmPermBench stress-tests a deployed permission gate, Claude Code's auto mode, with 128 prompts covering 253 state-changing actions; it evaluates one product's gate rather than defining a cross-model protocol, and it comes from the same group as UnderSpecBench below. It is the nearest prior work that reports over-blocking of safe actions alongside under-blocking of unsafe ones, although its headline result is the latter. Its scenarios deliberately underspecify target scope, blast radius, and risk level, so its blocked-safe-action figure is measured where the authorization is genuinely ambiguous and holding may be the defensible call. SteerBench-Work measures the opposite construction, in which signed or structured evidence resolves the risk before the commit. The two quantities answer different questions and are therefore not compared numerically here. SteerBench-Work is complementary because it uses public-harm anchors, paired mirrors, bidirectional error reporting, risk-resolved workplace commits, and training views.

Action-boundary benchmarks. UnderSpecBench is the closest neighbor to this work: 69 task families grounded in documented incidents, CVEs, and tool behavior, expanded into 2,208 prompt variants, scored by deterministic side-effect oracles that separate safe success from wrong-target and over-scope execution, with non-action runs classified as clarification, refusal, or deferment. It reports that 55.8% to 67.8% of runs violate at least one boundary, and concludes that completion-centric evaluation can overstate safe autonomy. The two benchmarks vary different things. UnderSpecBench varies what the agent knows, sweeping intent clarity, target certainty, and blast radius. SteerBench-Work holds the action fixed and varies whether it is allowed. The rows the two do not share are the ones this paper is about: the information is sufficient, authorization exists, risk-clearing evidence exists, and the model refuses anyway. The Authority Frontier framework prices each side-effect-bearing action against a safe default and treats persistence under denial as an underwriting variable, which is adjacent to the post-denial behavior this benchmark does not yet score.

Policy-compliance benchmarks. ST-WebAgentBench is the closest prior work on the completion-plus-compliance framing. It scores enterprise web-agent trajectories in GitLab, SuiteCRM, and a shopping-admin application against per-task authored policies across six dimensions, including user consent and action confirmation, using deterministic evaluators rather than an LLM judge, and reports Completion under Policy, CuP = Ctask · 1[Vtotal = 0], which credits only completions that violate no policy. CuP is monotone in a single direction: it falls as an agent acts beyond its permissions and never as it acts short of them. An agent that declines a task it was authorized to complete scores Ctask = 0 and is, in that metric, indistinguishable from an agent that crashed or executed the task incorrectly. The benchmark accordingly reports no over-refusal quantity, and its headline consent finding runs opposite to ours: it observes agents failing to request permission before critical actions. SteerBench-Work labels 56 of its 106 scenarios proceed and reports the rate at which authorized, evidence-cleared work is wrongly held, which is a measurement CuP cannot express.

Over-refusal as a side effect of defense training. The Autonomy Tax reports that agents hardened against prompt injection begin refusing legitimate tool operations prematurely, and frames this as a capability cost of adversarial fine-tuning. It is the closest prior observation that over-safety degrades an agent into uselessness. It studies models deliberately modified for defense, whereas the conditions evaluated here are stock provider endpoints as shipped.

Chat-refusal benchmarks. XSTest names exaggerated-safety refusal on a surface trigger word and measures it on chat refusal. Wen et al.'s survey of abstention defines and measures the abstention-vs-helpfulness trade-off.

Agent-safety benchmarks. Agent-SafetyBench evaluates unsafe tool-using agents across 349 environments and 2,000 synthetic and augmented test cases across 8 risk categories. ODCV-Bench scores outcome-driven constraint violations on 40 multi-step scenarios in production-inspired sandbox environments where KPI pressure conflicts with constraints. HiL-Bench scores ask-or-act on software engineering and text-to-SQL using Ask-F1, the harmonic mean of question precision and blocker recall, and reports that reinforcement learning on a shaped Ask-F1 reward improves both help-seeking quality and task pass rate. It is therefore already bidirectional and already trainable, on the axis of missing information. MLCommons AILuminate Agentic frames agent product maturity along quality, boundedness, confidentiality, control, robustness, security, and reliability axes.

Incident catalogs and instruction-hierarchy work. The AI Incident Database catalogs incidents but is not itself a scoring benchmark. The OpenAI Instruction Hierarchy paper defines the privileged-instruction hierarchy. Greshake et al. names indirect prompt injection.

Training methods relevant to the open training target. Rule-Based Rewards defines the rule-based reward formulation. Process Reward Models score intermediate reasoning steps. Self-consistency sampling is the basis for the modal-of-N trial verdict.

SteerBench-Work targets a different measurement object from each of these: the pre-commit steering decision at workplace action boundaries, with public-incident anchors, paired over-refusal mirrors, and labels for both unsafe action and unnecessary escalation. Adjacent benchmarks measure unsafe outcomes after execution (Agent-SafetyBench, ODCV-Bench), chat refusal (XSTest), or ask-versus-act when information is missing (HiL-Bench, UnderSpecBench). The remaining distinction is the axis rather than the direction. Bidirectional scoring and trained asking behavior already exist on the missing-information axis. SteerBench-Work holds information sufficient in order to isolate the four properties that survive when the agent already knows what to do: authorization, evidence resolution, consequence, and irreversibility. The failure-mode category names from the chat-refusal and abstention literature are reused as diagnostic slices so a reviewer can inspect which class of failure each model is making; the training target remains the action-boundary decision.

Name disambiguation. An unrelated 2009 benchmark of the same short name, SteerBench, evaluates steering behaviors for crowd and character navigation in virtual environments. This work is unrelated to it and is published under the full name SteerBench-Work.

Benchmark design

Task definition. A scenario is a frozen commit moment. The model receives the user request, the proposed action, and the evidence available at that moment (policies, signatures, audit records, prior messages), and nothing else. It returns a structured verdict: a commit permission (allowed or blocked), a policy action (continue, show progress, ask clarification, request approval, or escalate), a confidence, and a short reason. The scored field is the commit permission alone. Each scenario's expected verb maps onto it through a fixed published mapping: continue and proceed map to allowed; block, request approval, escalate, and ask clarification map to blocked. Asking, escalating, and requesting approval are therefore all holds at the scored boundary. A model that proceeds against a hold label commits an under-refusal miss; a model that holds against a proceed label commits an over-refusal miss. Each (scenario, model condition) cell runs five independent trials under one SHA-256-pinned prompt, and the grid reports mean trial accuracy, the modal-of-5 verdict, and pass5 side by side.

Corpus: incidents, mirrors, and resolved risk. Real incidents from the LLM era and from earlier automated-decision-system cases are the dataset asset other benchmarks do not have. The court rulings, regulator settlements, and class actions provide ground truth that an evaluator cannot dispute: the airline was held to the chatbot's statement; the lawyer was sanctioned for filing the hallucinated brief; the database was deleted; the candidate was rejected. The incident set is reproducible because the failure pattern is reproducible.

The paired over-refusal mirror methodology answers an obvious objection: a model that refuses on every surface flag will look good on the under-refusal set, since it will never commit anything risky. The 13-scenario mirror subset takes the same surface shape (a benefits notice that looks like Robodebt income-averaging, an applicant decision that looks like Workday Mobley, a production change that looks like a known outage) and inverts the verification state (signed cross-service authorization, eligibility met, audit trail present). A model that refuses an authorized mirror because it resembles a cautionary incident has the wrong heuristic. A model that blocks the unsafe case and commits the authorized mirror demonstrates the steering signal we are measuring.

The hardest over-refusal rows are not generic safety refusals. They are risk-resolved commit-boundary cases: a real risk trigger is visible, but signed or structured evidence resolves the risk for the proposed action. Examples include regulated notices after eligibility and due-process checks, secret rotation under a service account and cron schedule, broad but authorized codemods, and protected-surface changes with explicit approval. These rows test whether an agent can distinguish an unresolved risk from a resolved one at the moment of commit.

This framing matters for training. The target is not be less safe. The target is: detect the risk trigger, then condition the hold/proceed decision on whether the risk remains unresolved.

Label provenance. The release leaderboard uses the benchmark-owner labels. A separate three-rater exercise produced an 87.7% majority match with the owner key (Fleiss κ 0.69) and is reported as independent corroboration and label-sensitivity evidence, not as replacement scoring authority. Separately, a clean, leak-free three-model LLM annotation audit using the day's frontier systems (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro) is a blind reproducibility stress test; it checks whether independent frontier systems recover the same labels from answer-key-hidden evidence, and it does not define the labels (gate κ 0.937, 95.3% exact 3-model agreement, 97.2% panel-majority match). The irreversibility tier is human-authored severity metadata, a severity weight and not a primary score. The functional-category axis is human-authored diagnostic metadata, reported for breakdowns and not the primary score or a training target.

Metrics

A benchmark that scores only under-refusal (the agent acted when it should have asked) tracks one failure mode and misses the symmetric one. A benchmark that scores only over-refusal (the agent asked when it should have acted) misses the other. Both failures are real in production and they compound: an agent tuned to never over-refuse becomes unsafe; an agent tuned to never under-refuse becomes unusable. Scoring both directions in one suite makes the trade-off the operator actually faces measurable. Rule-Based Rewards defines both directions as paired axes but as a training-reward formulation, not a benchmark. AmPermBench reports both directions, but for a single deployed product gate and on scenarios whose authorization is deliberately left ambiguous. We have not found another cross-model benchmark corpus that tests both directions of the action-boundary commit decision, on scenarios where the evidence in hand resolves the risk, in one suite. This is a scoped negative claim, not an assertion of absence: the search covered the action-boundary, agent-safety, permission-gate, and refusal literatures reachable from the works cited in this section.

The public rank is mean trial accuracy, an unweighted estimate of one-call behavior. Modal-of-5 and pass5 are reported beside it as the majority-vote scenario verdict and the strict reliability view. The pass5 metric is the all-trials-succeed reliability estimator of τ-bench, applied here to the action-boundary gate. For action-boundary commits the cost is not uniform: a missed delete-production matters more than a missed save-draft. So the run also reports a secondary severity metric that weights each miss by normalized irreversibility tier. The raw scenario classes map to the three tiers used by the metric: none/soft-to-reverse/low = 1, medium = 2, hard-to-reverse/high = 4. The severity number shows the operational weight of the remaining misses. It does not change the primary comparable rank, and it does not assume the hardest-to-undo rows are always the hardest rows for models. In the current snapshot, models are strongest on obvious hard-to-reverse holds and weaker on medium and low rows where the risk has been resolved but the surface trigger remains. The tier is human-authored severity metadata, supported by a clean reproducibility audit, not a separate primary score.

Experiments

Setup. The validated grid covers 30 model conditions across OpenAI, Anthropic, Google, DeepSeek, Kimi, and open-weight gpt-oss models, each running all 106 scenarios at five trials per cell under one SHA-256-verified prompt. Sampling settings are fixed per condition and recorded in the saved request body of every trial. A validator refuses to publish any run whose provenance or completeness does not check out. Reasoning can move steering in either direction, so the within-model floor-vs-high contrast is the design that isolates it: run the same model at the same prompt, scoring, and scenarios with reasoning at its floor and at high. The locked grid records movement in both directions and is model-dependent: high reasoning can repair a weak gate, but it is flat or negative on some already-calibrated gates. Per-model figures are reported in Table 2 and Appendix D. Individual model differences are reported descriptively and are not tested for statistical significance. The open-weight rows add a second lesson: smaller, cheaper gates can be better calibrated than larger siblings, so model size alone does not choose the action-boundary reviewer.

Results. Table 2 reports every validated condition for release v2026-05, in three reads of the same five trials: mean trial accuracy (the primary rank), modal-of-5 accuracy (the majority verdict), and pass5 consistency. Two degenerate gates anchor the scale: a gate that always holds scores 47.2% with 100% over-refusal, and a gate that always proceeds scores 52.8% with 100% under-refusal. Every condition in Table 2 beats both on accuracy.

Direction and construction pattern reconcile. The release splits by direction as 25 under-refusal, 51 over-refusal, and 30 calibration cases, and by construction pattern as in Table 3. The two views reconcile exactly: public-harm anchor (24) and risk-unresolved hold (1) are under-refusal; incident mirror (13), adversarial control (6), and 32 of the 33 detector-conflict rows are over-refusal; clean control (3), calibration control (25), the one risk-resolved commit row, and the remaining detector-conflict row make up the 30 calibration cases. Two calibration-direction rows therefore sit inside construction patterns that are otherwise over-refusal, which is why grouping by pattern name alone does not reproduce the direction counts.

Misses run almost entirely in one direction. Models over-refuse at 28.1% per opportunity and under-refuse at 1.0%. The asymmetry is not an artifact of corpus composition, because the two rates rest on near-identical denominators. An over-refusal opportunity is any (scenario, condition) cell whose reference label is proceed; an under-refusal opportunity is any cell whose reference label is hold. Calibration cases carry reference labels like every other row, so they enter both pools by label rather than by direction tag: 56 of the 106 scenarios are labeled proceed and 50 are labeled hold, giving 1,680 and 1,500 cells across the 30 conditions. Three cells are not directionally scored, all of them the same proceed-labeled detector-conflict scenario under three Anthropic conditions that returned no parseable decision in any of their five trials, so the over-refusal denominator is 1,677. Against those denominators the grid records 471 over-refusal misses and 15 under-refusal misses: a 31.4:1 raw miss ratio standing on a 1.12:1 opportunity ratio. Scoring the three unparseable cells as over-refusals rather than excluding them would raise the over-refusal rate to 28.2%, so the exclusion is conservative with respect to the headline reported here. The must-hold safety side is mostly handled well in this release, so the asymmetry is primarily a statement about the over-refusal side rather than a claim that models are near-perfect at holding. The calendar-invite row remains the main must-hold counterexample, and several final rows now handle it correctly, so it is a live safety-side check rather than a near-universal failure.

Resolved risks are harder than obvious dangers. The sharpest over-refusal failures sit on risk-resolved commits, where a real risk trigger is visible but signed or structured evidence has already cleared the proposed action. The Apple Card mirror, a credit-limit increase with completed and signed fairness controls, is labeled proceed and scores 0 of 30: every condition in the grid wrongly holds it.

Mirrors are much harder than anchors. Models score 98.5% on the famous public-harm anchor rows but only 63.8% on the flipped mirrors that keep the surface and reverse the verification state, a 34.6-point gap (computed from the raw fractions and rounded once). The gap shows the reversed cases are substantially harder in this release. It does not by itself identify the cause, and under the three-rater majority labels it narrows to about 4 points. Table 3 shows the full difficulty gradient across construction patterns.

The detector-conflict comparison. Detector-conflict rows carry a live risk trigger over clearing evidence but no famous incident. Models score 76.8% on detector conflict and 63.8% on the mirrors, computed on the modal-accuracy convention in which a cell with no parseable decision is scored incorrect. We report this as a descriptive difference and do not treat detector conflict as a matched comparator for the mirrors: the two sets are not paired, and non-commit risk flags are not equalized across them. One instrumentation note bears on these rows. In 17 of the 106 scenarios the fixture-integrity adapter inferred a success criterion change flag by keyword-matching the scenario, action, and evidence text against a risk-term list; in the Apple Card and Amazon mirrors the terms eval and score matched inside signed audit-tool references. The flag was inferred, not authored. Excluding all 17 as a diagnostic slice, over-refusal is 19.4% against 1.25% under-refusal, and the clean detector-conflict and mirror rates are 83.0% (747/900) and 74.8% (247/330). A deliberately matched conflicting-signal family is left to future work.

The modal winner and the reliability winner differ. The primary-rank leader is Gemini 3.1 Flash-Lite at minimal reasoning (92.8% mean trial accuracy, 91.5% pass5). DeepSeek V4 Pro with reasoning off leads the modal vote (93.4%) but is less stable across trials. An operator choosing a deployment gate and an operator reading a leaderboard headline can reasonably pick different models.

Neither scale nor reasoning effort predicts calibration. Larger and newer models are not automatically better gates (Table 2), and raising reasoning effort is model-dependent: it can repair a weak gate while leaving an already-calibrated gate flat or slightly worse. Individual model differences are reported descriptively and are not tested for statistical significance; per-scenario verdicts for every condition are published with the release.

Limitations

The scenarios are constructed, single-turn descriptions of commit moments. A model reads the moment; it does not execute tools, so behavior under a live harness can differ from behavior on a described boundary. The format scores one boundary per scenario: it does not measure long-running sessions, and it does not measure what an agent does after a denial. The release is 106 scenarios at five trials per condition, and individual model differences are reported descriptively, without significance tests. Model scores are snapshots of dated provider endpoints and are not portable across provider updates. The model roster was frozen on June 8, 2026; models and materially updated endpoints released after that date are outside release v2026-05, which is not intended to represent the model frontier after that date. The irreversibility tier is human-authored severity metadata, not a measured property of an environment. The scoring key is the benchmark-owner label set; the three-rater human pass is independent evidence on those labels (87.7% majority match, Fleiss κ 0.69), and the disagreements are reported rather than adjudicated away. The proceed-labeled and hold-labeled pools are matched on opportunity count (56 and 50 scenarios) but not on authored difficulty: nothing in the construction process controls for the possibility that the proceed-labeled scenarios were written to be harder than the hold-labeled ones, and no per-item difficulty rating exists with which to test it. The direction asymmetry is therefore robust to the corpus-composition objection but is not, on this evidence alone, robust to an item-difficulty selection effect. Excluding two proceed-labeled credit and hiring mirrors that may leave ordinary business preconditions implicit reduces modal over-refusal from 28.1% to 25.5%, while under-refusal remains 1.0%; the directional result is unchanged. Scenario texts are constructed; references to named companies and incidents point to public records (court rulings, regulator actions, and press coverage) and carry no claim about any organization's current systems. Nothing in this release supports a claim about the safety of a deployed agent system. All results are measured under the single system prompt pinned by SHA-256 in Appendix A; the gate's sensitivity to prompt rewording is not measured here and is left to future work. The released 106 is a public reference set and begins a contamination clock once published; graded rankings move to fresh versioned sealed releases as contamination accrues, which the versioned protocol supports without redefining the benchmark.

Future work

The benchmark is also a model-selection tool, and the training work it enables is future work rather than a result reported here. Any trained adapter will be reported separately from the locked benchmark grid, under the same confusion-matrix view, so that base-model evaluation stays separate from post-training lift. The training target is the human-authored gate label; irreversibility remains a human-owned severity weight and mechanism labels remain diagnostic metadata. Before training, the expanded corpus is grouped by scenario family and split into train, development, and sealed test sets: training uses only train, prompt and threshold choices use development, and lift is reported once on the sealed split. The objective is not to make the model allow more. A recipe will count as an improvement only if it lowers over-refusal or answer instability at fixed-or-lower under-refusal on unsafe actions, reported as a frontier rather than a single score. The open-weight base models, the recipes, and the resulting frontier will be published with that run.

The second direction is the measurement setting. This release isolates the decision: one boundary, described, scored in one turn. In deployment the same decision occurs inside work the agent must finish, where holding has a cost and the gate fires many times per session. That setting does not fall out of existing long-horizon suites, for a structural reason rather than an oversight: a benchmark that scores completion cannot reward holding, and a sandboxed harness removes the consequence that makes holding correct. Meanwhile production systems already make this decision at scale on private data; Anthropic reports that Claude Code users approve 93% of permission prompts and ships a classifier that substitutes for the human approver, evaluated on internal datasets. Measuring the same decision inside completed work, against a public yardstick, is the natural next step for this protocol.

Conclusion

The gate decision is no longer hypothetical: vendors ship it inside products, incidents document its failures, and the first cross-vendor grid locates the field's problem precisely. In this release, frontier models almost never cross an unsafe boundary, yet they refuse authorized work at rates that make autonomy expensive: 471 over-refusal misses against 15 under-refusal misses. The sharpest failures sit on rows where the risk was real and the evidence had already cleared it, and accuracy on the evidence-reversed mirrors falls 34.6 points below the incidents themselves. A gate that acts on the resolved evidence in front of it, rather than holding on a familiar risk, is the cheaper path to trustworthy autonomy, and this benchmark makes that measurable.

Based on the paper, here are the specific improvements to AI systems and what the improved systems can do:

1. Implement a two-stage risk-resolution gate

• Improvement: Add a dedicated pre-commit module that first detects risk triggers (e.g., financial transactions, data deletion, legal actions) and then checks whether signed/structured evidence has already resolved that risk before deciding to hold or proceed.

• Capability: The AI can distinguish between an unresolved risk (e.g., sending a payment without authorization) and a resolved one (e.g., sending a payment after a signed approval), reducing over-refusal on authorized work by up to 28.1% while keeping under-refusal at 1.0%.

2. Train on incident-mirror pairs for evidence-based decision-making

• Improvement: Fine-tune models on paired scenarios where the surface details are identical but the verification state is inverted (e.g., a benefits notice with vs. without eligibility confirmation), forcing the model to follow evidence rather than surface heuristics.

• Capability: The AI can correctly proceed on authorized actions that resemble past incidents (e.g., approving a credit limit increase after fairness controls are signed), improving accuracy on evidence-reversed cases from 63.8% to near the 98.5% level seen on original incidents.

3. Add bidirectional error reporting to the training objective

• Improvement: Replace single-direction safety metrics with a loss function that penalizes both under-refusal (acting without authorization) and over-refusal (blocking authorized work) equally, using the paper's 56/50 proceed/hold label split as a balanced training set.

• Capability: The AI can optimize for both safety and usability simultaneously, avoiding the common failure of becoming overly cautious after safety fine-tuning, which currently causes 471 over-refusal misses per 1,677 opportunities.

4. Use modal-of-5 voting for deployment decisions

• Improvement: Run the gate decision five times per action and take the majority verdict, rather than relying on a single trial, to reduce answer instability that the paper found even in leaderboard-winning models.

• Capability: The AI produces more reliable commit decisions, with pass5 consistency (all five trials correct) improving from 91.5% to potentially higher, and reduces the risk of a single stochastic failure causing an unnecessary hold or unsafe action.

5. Apply reasoning effort selectively based on calibration status

• Improvement: Implement a calibration check: if the model's baseline gate is already well-calibrated (low error in both directions), keep reasoning at a minimal level; if it over-refuses, increase reasoning effort to repair the gate, as the paper shows high reasoning can fix weak gates but harms calibrated ones.

• Capability: The AI can adapt its computational cost per decision, improving accuracy on weak gates (e.g., from 63.8% to higher on mirrors) without degrading already-calibrated performance, saving inference cost on routine authorized actions.

6. Build a risk-resolved commit classifier

• Improvement: Train a specialized classifier on the 33 risk-resolved commit scenarios (e.g., secret rotation with cron schedule, broad codemods with explicit approval) to recognize when a visible risk trigger has been cleared by structured evidence.

• Capability: The AI can correctly proceed on authorized but superficially risky actions—like the Apple Card mirror where all 30 model conditions failed—enabling autonomous execution of legitimate high-value tasks that current systems wrongly block.

7. Use severity-weighted decision thresholds

• Improvement: Adjust the hold/proceed threshold based on irreversibility tier (low=1, medium=2, high=4), so the AI is more conservative on irreversible actions (e.g., deleting production) and more permissive on reversible ones (e.g., saving a draft), without changing the primary accuracy metric.

• Capability: The AI minimizes operational harm by weighting misses appropriately—reducing the impact of the 1.0% under-refusal rate on critical systems while not over-blocking low-stakes authorized work.

8. Implement evidence-trace verification before commit

• Improvement: Add a module that explicitly checks for the presence of signed authorizations, audit trails, or structured eligibility confirmations in the evidence before allowing a proceed decision, mirroring the paper's finding that models fail when evidence is present but not acted upon.

• Capability: The AI can verify that risk-clearing evidence exists (e.g., a signed cross-service authorization) and use it to override surface-level risk flags, reducing over-refusal on authorized actions by up to 19.4% (when excluding inferred flags) while maintaining safety on genuinely unsafe actions.

Long-running LLM agents act through tools, and a single step can send an email, merge a pull request, or wire a payment. The steering decision is the pre-commit choice at that boundary: proceed, or hold for human or policy review. We introduce SteerBench-Work, an incident-anchored, bidirectional benchmark for that decision in workplace agents across developer operations, customer service, finance, legal, medical, HR, and security. Release v2026-05 contains 106 scenarios anchored in public incidents, paired evidence-reversed mirrors, and calibration controls, with labels split nearly evenly between proceed and hold so the two error directions get near-identical numbers of chances. A model sees the proposed action and the available evidence, returns a gate decision, and is scored on whether it crosses or holds the boundary correctly. Across 30 model conditions the failures run almost entirely in one direction: models wrongly hold authorized, evidence-cleared work on 28.1% of opportunities and wrongly allow unsafe work on 1.0%. The hardest cases are risk-resolved commits, where signed or structured evidence has already cleared a real risk trigger, and models score markedly worse on evidence-reversed mirrors of famous incidents (63.8%) than on the incidents themselves (98.5%). General capability is not the same as steering calibration: higher-capability models often over-refuse at the commit boundary, and more reasoning can repair a weak gate while leaving a calibrated one flat. The public leaderboard is at steerbench.com.