Think Before You Act: How JevAct Gates LLM Agent Actions

An agent’s action should pass through a gate that can check, compute, and send back before anything executes. JevAct splits that gate in two. code settles what tool records decide, a decision model answers only “what did the conversation actually say”, and a boolean rule combines them. It lifts all eight agents on τ²-Telecom, and it is unusually honest about where the gate stops working.

English voiceover, 2:43. The Chinese cut covers the same material.

The paper is arXiv:2610.38620, Think Before You Act: Decision Gate for LLM Agents (IBM SIL), 18 pages. Everything below was written after reading all of it, and every number, threshold and caveat traces back to a table or a paragraph there.

1. The problem: propose, execute, hope

A ReAct loop has one default behaviour: whatever the model emits, the harness runs. That is tolerable when a wrong answer can be corrected next turn. It is not tolerable for three kinds of action (§1):

Existing methods each cover part of this. ReAct interleaves reasoning with acting. Self-Refine and Reflexion critique after the fact and try again. A family of runtime-enforcement systems (§2.2). GuardAgent, compiled company-policy guards, AgentSpec, ShieldAgent, CUGA, ToolSafe, PolicyGuard, ToolGuard-style verifiers, ToolFence, CaMeL, puts explicit controls around execution. LLM-as-a-judge asks a second model to review (§2.3).

What is still missing is the narrow thing this paper targets. before this one action executes, decide with a rule that can be run and a probability that can be thresholded. A generative reviewer produces an explanation that must itself be interpreted before a gate can act on it, and doing that with a frontier model for every write is expensive.

2. The idea: two kinds of evidence, split by what can settle them

Fig. 1. The agent still proposes; the gate only decides execute or send back.

JevAct is a harness around the ReAct loop. It does not choose actions; it verifies them. The design move is splitting “is this action correct?” into two questions with different evidence:

• Facts the tool records settle: how many hours ago the booking was created, the free-baggage allowance, whether a segment has already flown, whether an amount exceeds a balance. These go to code, because arithmetic and date comparison are exactly what a decision model is worst at.
• Things only the conversation can settle: did the user ask for this, did they confirm, was a troubleshooting step tried. These become yes/no questions for a decision model.

A boolean rule combines the two. That is the paper’s real claim, namely that deterministic facts should stay out of the probabilistic model and conversation-dependent conditions should stay out of the code.

3. The four components

3.1 The decision loop

Four stages per step:

1. Propose. The LLM reads history h_t and emits a message m holding user-facing text and, when it wants to act, tool calls.
2. Trigger. A write call routes to the write gate; a hand-off routes to the hand-off check; anything else executes as in ReAct.
3. Check. Returns pass or flag, with a feedback string f when flagged.
4. Commit or revise. A passing message is executed and appended. A flagged message is discarded: never executed, never stored, and the agent proposes again with f attached.

Two details matter, both in §3.2.

• Feedback is temporary. It is visible only to that one re-proposal and never enters the conversation history, so critiques do not accumulate in the context.
• The budget is hard. R = 2 for the write gate, R = 1 for the hand-off check. Once the budget is spent, the next proposal is committed without another check. A gate can never block an agent indefinitely, and it adds at most R extra LLM calls (§3.2).

3.2 The compiler

Offline, once per domain, an LLM (Claude Opus 4.6 in this work) turns the policy into four artefacts: yes/no questions, boolean rules, code that computes facts, and a hand-off checklist. The six steps and their inputs are Table 6, and the full prompts are Appendix A.1 (§3.3).

3.3 The judge

Kev, an open-source Jev-type decision model (§3.4). Given a prompt and questions with fixed options, it returns a probability per option in one forward pass, generating no text. Questions sharing a prompt share that pass. So the cost of a check does not grow with the number of questions. The paper uses Kev-27B, a fine-tune of Qwen3.8-27B, with no further training. Because Kev is a different model from the agent, the agent never judges its own proposal.

3.4 The checks

A write gate, a hand-off check, and. On BFCL, where no written policy exists, two generic checks.

4. Compiling a policy: six steps, two filters

Fig. 2. Step 5 is a local filter, not an LLM call; both filter thresholds were set once on Airline and reused.

4.1 Questions: why “atomic” is the whole point

The compile prompt describes the reader precisely: a small model that reads, cannot compute, and answers yes/no with a calibrated probability. So each question must test one fact and be answerable from the conversation; arguments appear as placeholders ({reservation_id}) that code fills with the proposed call’s values at run time. The prompt forbids asking the judge to compute anything. Arithmetic is step 3’s job.

Airline yields 86 questions across six write tools (12-21 each); Telecom 75 (Table 1). A slice for cancel_reservation:

• reservation_id_correct. Does {reservation_id} match a reservation confirmed by tool results to belong to this user?
• no_flown_segments. Do the records show zero flown segments in this reservation?
• cancellation_eligible_24hr. Was the booking made within the last 24 hours, per the records?
• at_least_one_cancel_condition_met. Is at least one cancellation condition met (24 h, airline-cancelled, business cabin, insurance)?
• user_requested_cancellation. Did the user actually ask to cancel, or merely inquire?
• user_confirmed_cancellation. Did the agent list the details and the user explicitly confirm?

Each question carries bad_if: which answer means a violation. That is how the gate knows the polarity instead of guessing from wording.

4.2 Rules: why a list of questions is not enough

Policies are full of “any one of these is enough”. An or that independent questions cannot express. Step 2 asks the model to write one boolean expression. The compiled cancel_reservation rule:

1user_id_obtained and reservation_id_correct and reservation_retrieved
2and no_flown_segments and cancellation_reason_obtained
3and at_least_one_cancel_condition_met
4and user_requested_cancellation and user_confirmed_cancellation

Note that it keeps the summary eligibility question and drops the four single-condition ones, which the summary already covers. At run time each id is true unless Kev gives a confident no, and the expression is evaluated with eval and no builtins.

4.3 Code: report only what the records make certain

derive_<tool> receives the proposed args and every earlier call with its parsed result, and returns computed facts plus a violations list.

The constraints are strict:

• standard library only (json, math, datetime), executed with a builtin/import whitelist;
• an exception counts as no violation. Better to pass a bad call than to block a good one;
• a violation is reported only when the records make it certain. Missing record, or a rule that depends on what the user said? Leave it to the judge.

A real excerpt, for update_reservation_baggages:

1free_per_pax = _free_bags_per_passenger(membership, cabin)
2total_free = free_per_pax * num_passengers
3expected_nonfree = max(0, new_total - total_free)
4if new_nonfree != expected_nonfree:
5    violations.append(
6        f"nonfree_baggages should be {expected_nonfree} "
7        f"({membership}, {cabin}, {num_passengers} passenger(s): "
8        f"{total_free} free; new total {new_total}), call has {new_nonfree}")

4.4 Repair: one round, bugs only

The step-3 code is run over write calls from train-split logs. Up to 16 flagged calls (one per tool, first violation) are fed back with the records the code read. no gold labels, and the compiler decides whether each reported violation is real, then fixes the code.

This round cuts false alarms on held-out Airline tasks from 11.3% to 1.4% of correct write calls (Appendix A.1, step 4). The bugs are specific: reading an error string like "Error: User not found" as a user record; using a record that a later tool result had superseded.

4.5 Filter: drop the questions the judge cannot read

Kev answers every compiled question on every write call in the development logs. Any question that fires. A confident violation, on more than 10% of its tool’s logged calls is dropped: real violations are rare, so a frequently firing question is either unreadable from the conversation or not required by the policy.

Thresholds were set once on Airline and reused unchanged (Appendix A.1, step 5): Airline drops 17 of 86 questions, Telecom 5 of 75 (over 175 logged write calls). A dropped question is never reported and counts as yes in the rule.

4.6 The hand-off checklist: one instruction does the work

The hand-off check asks whether the agent gave up too early, so the compiled artefact is: the problem types the policy troubleshoots, and for each, the steps that must be tried before giving up. Each step carries a settled question that asks whether it was tried, or whether the conversation established that it does not apply.

The most important line in the compile prompt:

A condition that was never discussed is not ruled out.

Without it, “enable roaming” compiles to “…or the user is not travelling”. Which looks settled although nobody ever asked. With it, the checklist gets its own step: ask whether the user is travelling.

The checklist has a different filter (Appendix A.1, step 6): steps flagged on more than 5% of hand-offs in solved training episodes are dropped, since a correct hand-off should pass. Calibrated on 74 train-split tasks (217 hand-offs, 36 inside solved episodes), this removes seven steps: airplane mode, APN settings, SIM status, line suspension, overdue bills, contract end date, and resuming the line after payment.

5. The write gate: three kinds of evidence, one threshold

Fig. 3. Only P(yes) ≤ 0.05 counts as no; the feedback tells the agent whether to fix the call, ask the user, or explain why it cannot be done.

Three steps before every write call (§3.5):

1. Code reads the proposed arguments and earlier results and returns the violations the records establish.
2. Judge answers that tool’s questions, given the policy, the conversation, the tool results (dates annotated with their offset from now, option lists pre-ranked) and the proposed call.
3. Rule. A code violation, or an unsatisfied boolean, means flag.

The threshold semantics deserve their own sentence: only P(yes) ≤ 0.05 counts as no; everything else counts as yes. That is deliberately conservative. The gate prefers to let a bad call through over blocking a good one. The paper does not pretend these probabilities are calibrated; it treats the threshold as an engineering choice to be validated inside the loop (§3.1, §3.5).

A flag lists the established violations and the failed questions, and asks the agent to do one of three things: correct the call, ask the user for the missing information, or explain why the request cannot be fulfilled. An agent that still believes it is right may submit again; after a second flag, the feedback directs it to reply to the user instead of calling the tool again.

On Airline the gate is a light touch: Opus 4.6 sends back 0.08 drafts per run, Qwen3.8-27B 0.03 (§5, Table 9).

6. The hand-off check

Fig. 4. Identify the problem type, then each required step’s settled question; one open required step means flag.

Two triggers: a transfer call, and a give-up message (“I can’t help with that”). A keyword filter finds candidates, then the judge confirms.

In one request Kev answers three things: which compiled problem type the user has (or other), whether the user says it is fixed, and every step’s settled question. The rule:

• a step is required when P(the problem needs it) ≥ 0.5;
• it is open when P(settled) ≤ 0.5;
• if the problem is not fixed and at least one required step is open → flag, and return the open steps as feedback.

The hand-off check allows one revision (R = 1) and runs only on Telecom. Disabling it on Airline is evidence-based: offline, it flagged 26% of hand-offs in otherwise solved episodes (§4.5).

7. No policy at all: the two BFCL checks

BFCL multi-turn has no written policy and its user turns are fixed, so nothing can be compiled. The paper follows BFCL’s own loop, with no system prompt and up to 20 steps per user turn (§4.1). JevAct uses two generic checks:

• End-of-turn. Fires when the agent ends its turn while a requirement-bearing tool error (wording like “needs to”, “must”) is unresolved. Kev selects the tool that satisfies the requirement from the tool list, or none; at P ≥ 0.6 for a tool, the feedback directs the agent to call it rather than go back to the user.
• Requested-action gate. Before each state-changing call, asks whether the user requested it or a tool error required it. At P(yes) ≤ 0.05 the call is flagged as unrequested, and the feedback limits the agent to what was asked or required.

8. Experimental setup

The metrics need a note:

• pass@3. Share of tasks solved in at least one of three runs;
• pasŝ3. Share solved in all three runs (repeatability);
• pasŝ1. Mean per-run success.

Main results use three runs per task, and the metric definitions are §4.4. The component ablations are single-trial pasŝ1, so differences between rows must be read cautiously.

9. Main results

Fig. 5. All eight agents improve on Telecom; Opus 4.6 goes from 39.5% to 80.7%. Numbers from Table 2.

9.1 Telecom: the consistent result

All eight agents improve on both metrics: on average pass@3 +16.0 points and pasŝ3 +16.9 points (Table 2, Fig. 1).

Even the strongest ReAct baseline, Opus 5.5 at 74.6% pasŝ3, reaches 82.5% (§5). The pattern is consistent with the hand-off check targeting a common failure mode (premature abandonment), but the paper is careful: the comparison evaluates the whole harness and does not isolate the hand-off check’s contribution.

9.2 Airline: small and mixed

pass@3 improves for four of six agents, is unchanged for one and falls for one; pasŝ3 improves for four and falls for two. GPT-4.1 goes 66.0 → 82.0, Opus 4.6 goes 84.0 → 92.0.

Two things to keep in mind: with 50 tasks, one task is 2 points; and the write gate barely intervenes for these agents (0.03-0.08 drafts per run).

9.3 BFCL: more repeatable, not broader

All nine agents improve pasŝ3 (+0.1 to +9.6 points), while pass@3 improves for four and declines for five (0.2 to 3.6 points).

This is the easiest result to misread: “solved in all three runs” is not “solves more tasks.” The more defensible reading is that the gate makes behaviour on some tasks more repeatable rather than widening capability.

10. Ablations: three counter-intuitive findings

10.1 Executable checks do more than the judge

Fig. 6. One change per row, pasŝ1. Numbers from Table 3.

Three readings:

1. Programmatic state checks account for most of the gain (68.0), while the judge’s questions alone reach 57.1.
2. A smaller judge costs real accuracy (58.0). Judge choice matters in this setting.
3. The placebo gains nothing: flagging every write unconditionally once is still 50.0. So the gain comes from sending back the calls that were judged wrong, not from the extra attempt itself.

Point 3 needs an important caveat. With gpt-oss-120b as the agent, the same unconditional re-check lifts pasŝ3 from 32.0 to 42.0. As much as the full gate; random flags at the gate’s rate do the same. Whether the judge’s verdict matters beyond the revision opportunity therefore depends on the agent: for GPT-4.1 the full gate reaches 40.0 versus 28.0 for the unconditional re-check (paired 95% CI of the gain: +2.0 to +26.0 points), while for gpt-oss-120b they tie.

10.2 Any judge helps; the cost differs by one to two orders of magnitude

Table 4 swaps the judge behind the same Airline write gate (§5.1):

Every judged configuration improves on its own ReAct baseline by 9.4-18.7 points, so the framework is not betting on one judge architecture. Kev’s position is cost: success on par with generative judges at one to two orders of magnitude less.

10.3 A bigger judge is not reliably better

Table 5 swaps 27B for 9B and 0.8B while keeping checks, thresholds and dropped-question lists fixed (§5.1):

• On Airline, GPT-4.1 does prefer 27B (65.3 vs 59.3 / 58.0), but the other agents show no consistent preference;
• on Telecom the smaller judges often match or beat 27B: Opus 4.6 rises from 93.6 to 99.1 with 9B, DeepSeek from 85.7 to 98.2 with 9B;
• replaying the same 609 hand-off states, Kev-9B agrees with Kev-27B on 97.0% of decisions and Kev-0.8B on 90.6%.

The replay also holds a less flattering detail. Kev-0.8B flags less (342 against 371), catches 299 of 533 required fixes, and marks 11 failed episodes as fixed that were not (Appendix A.2). So the smaller judge scoring higher on Telecom is not evidence that it judges better; there is an agent reacting to its feedback in between.

Two mechanisms explain the rest:

• Input length. 0.8B/9B are validated to 8,192 tokens, 27B to 65,536. On Airline 41% of write-check inputs exceed 8,192 tokens; on Telecom only 4% of hand-off inputs do. On long inputs Kev-9B’s AUROC is 0.61 against 0.85 for 27B.
• Threshold calibration. Recalibrating Kev-9B from 0.05 to 0.10 on short inputs raises catches from 8 to 16 (27B catches 21) with no extra false alarm.

So the conclusion is that judge size is not the lever: judge choice depends on context length, threshold, and the kind of verification; and downstream success also depends on how the agent reacts to feedback. Which is why Table 5 must not be read as a ranking of judge accuracy.

11. What it costs

Table 9 reports tokens per run (thousands, Appendix A.5):

• Agent side barely moves: Airline GPT-4.1 goes 58.4k → 60.1k input; Telecom rises 10-35% because more drafts are sent back (GPT-4.1 130.6k → 167.3k).
• Judge side is small: 3.2k–24.1k input tokens per run, depending on benchmark and agent.
• Drafts sent back: 0.03-0.39 on Airline, 0.10-1.91 on Telecom, 0.02-0.13 on BFCL.

Self-hosted latency (A100-80GB, bf16, four concurrent requests, seconds per request, Appendix A.6): Kev-0.8B 0.53, Kev-9B 1.22, Kev-27B 2.72, Qwen3.8-27B 4.02, Gemma-4-31B 6.10. Costed at $2 per GPU-hour at full utilisation.

12. Limits the paper states itself

This is the part worth reading twice.

1. It is sensitive to the user simulator. Swapping Telecom’s simulator from GPT-4.1 to gpt-oss-120b makes the gain disappear: pasŝ3 changes by −1.8 points (95% CI −7.9 to +4.4). The Telecom consistency result is entangled with simulated user behaviour.
2. The questions encode the policy, not the intent. Of 260 labelled development write calls, 101 are incorrect. And 42 of those break no compiled question, because the questions come from the policy while the error came from what the user actually wanted.

Fig. 7. The judge is one to two orders of magnitude cheaper than a generative one; yet 42 of 260 labelled write-call errors break no compiled question.

3. The ablations are single-trial. Tables 3-5 need cautious reading; the main results use three runs.
4. It evaluates the whole harness. Compiled checks, feedback opportunities and the budget are all in the comparison, so the results cannot be credited to the judge alone.

Two engineering dependencies: compilation uses Claude Opus 4.6, and the thresholds and dropped-question lists were fixed once with Kev-27B on Airline. A new domain means a new compile.

13. Where it sits

14. When to use it, when not to

Good fit

• the action is irreversible and the policy can be written as conditions (support, orders, change management);
• you have tool schemas and a slice of development logs. Compilation and both filters need them;
• you accept a conservative threshold, i.e. You would rather pass a few bad actions than block good ones.

Poor fit, or extra work needed

• policies that are genuinely judgement calls;
• long inputs (>8k tokens) with small decision models;
• expecting the gate to infer intent. The 42/101 gap says that has to be checked at the intent layer instead.

What the authors point at next: a common interface for proposed actions, tool state and execution outcomes so the gate is not tied to one ReAct loop; automating check derivation, validation, and the choice of thresholds and budgets on held-out traces; and evaluation that reports end-to-end latency, cost and safety alongside success and repeatability.

References

• Shoham, David, Akrabi, Bilgory, Levy, Marreed, Zeltyn, Shlomov. Think Before You Act: Decision Gate for LLM Agents. arXiv:2610.38620, 2026. Paper: arxiv.org/abs/2610.38620, PDF: arxiv.org/pdf/2610.38620.
• Related: τ²-bench (Barres et al., 2025), BFCL multi-turn (Patil et al., 2025), ReAct (Yao et al., 2022), Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), AgentSpec (Wang et al., 2026), ShieldAgent (Chen et al., 2025), ToolFence (Li et al., 2026), Jev (TypeSafe AI, 2026), Kev (Palmer, 2026), JevLoop (ZJUNLP, 2026).

Companion video: a Chinese and an English cut, walking through how the gate is compiled, how it decides, and where it fails.

• Previous
  SecLog, read closely: letting a delivery leave its own receipt
• Next
  Cognitive surrender: hand off the work, not the judgement