EPV research artifact

Prediction Is Not Permission

Statistical Execution Authority for AI Agents

Capable systems can propose and rank actions. They still need evidence before they are allowed to act.

In a multi-candidate agent pipeline, a source-level safety signal need not survive eligibility checks, scoring, and adaptive selection. This paper certifies the selected action of a frozen pipeline, then routes it to authorize, review, or abstain under registered conditional-risk bounds.

In plain language: an agent may suggest a good action, and the system may choose its most convincing suggestion, but neither step alone proves that the final action should be executed.

Read the 7-page paper Inspect claims Start reviewer check View public artifact

The selected-action permission gap

Planner ATool BAgent C

Frozen selectorSelected action

Post-selection evidenceEPV-Opt gate

Authorize Review Abstain

Inspect the mechanism

Reviewer mode

What is new, and what the evidence can support

EPV identifies selected-action authority risk as a post-selection object and introduces split-local EPV-Opt to certify and route the final selected action under pipeline-bound finite-sample risk bounds.

Proposal quality!=Selected-action evidence!=Execution permission

Registered claim

Candidate-level evidence does not determine selected-action risk.

Candidate count alone does not determine whether the final selected winner becomes more or less adverse.

Paper: Section 2; Section 4 Theorem 1 and replacement propositions; Section 6 RQ1.
Evidence: Heterogeneous selected-action identity, dependence-free replacement decomposition, and sealed K=1,2,4,8 MiniWoB study.
Artifact: Main paper and selected-action risk figure.
Full check: make sea-theory and make sea-v4-audit from the uploaded Code and Data Supplement.

Interpretation boundary

This does not claim that adding candidates always increases risk. The independent heterogeneous identity and the dependence-free replacement decomposition have distinct assumptions.

Registered claim

Post-selection local certification can route unsupported actions away from direct execution.

The final selected action is the object tested by EPV-Opt; score clearance alone is not authorization.

Paper: Sections 3, 4, 5, and 6 (RQ2/RQ3); labels fig:ps_epv, fig:sea_results, and tab:sea_v4.
Evidence: MiniWoB K=8 split-local result: direct 0.194, review 0.086, observed FAR 0.000, test UCB 0.008.
Artifact: Main paper, Figure 2, and sealed result figure.
Full check: make sea-v4-audit, make sea-v4-tables, and make verify from the uploaded Code and Data Supplement.

Interpretation boundary

Applies only to the frozen registered MiniWoB pipeline, task groups, adverse-action proxy, and IID split assumptions. It does not establish production harm prevention or universal agent safety.

Registered claim

Certificate validity is pipeline-versioned.

An unregistered candidate, registry, score, threshold, tie-rule, task-family, or learned-state change fails closed until recalibration.

Paper: Section 3 Decision and versioning, Section 6 RQ5, and Section 7 Limitations and Impact.
Evidence: Figure 2 fingerprint gate, mutation checks for candidate count, role registry, and learned weights, plus the negative AgentDojo V10 holdout.
Artifact: Main paper, Figure 2, and release boundary.
Full check: make verify from the uploaded Code and Data Supplement.

Interpretation boundary

Versioning is an invalidation rule, not a drift-robust, online, or environment-independent certificate. AgentDojo V10 does not confirm transfer.

MiniWoBK = 8EPV-Opt split-local

Direct0.194

Review0.086

Observed FAR0.000

Test UCB0.008one-sided 95% diagnostic

What this does not establish

Production safety
Legal or organizational permission
Positive independent AgentDojo transfer
Environment-independent or post-update certificate validity

Suggested reviewer path

Run the reviewer check in five stops

ClaimsIdentify what is new and select C1, C2, or C3.
EvidenceTrace each claim to a paper section and released artifact.
MethodFollow propose, select, certify, and route in order.
LimitsCheck the assumptions and explicit non-claims.
ReproduceRun the public sanity check, then use the full supplement.

1. Reality: the problem

A confident suggestion is not permission to change the world

Imagine an assistant that can send an email, edit a document, call an API, or submit a form. It can produce a fluent plan and report high confidence. The practical question is not only whether the plan sounds reasonable. It is whether the exact action selected by the whole system should be allowed to change external state.

A multi-source system may receive proposals from a planner, a rule, a tool, a critic, or a cached rationale. A source can look safe in isolation, while selection favors a harmful or unsupported candidate. The paper therefore studies the final selected action, not the confidence of an individual source.

Everyday situation	What can go wrong?	Paper's question
Many suggestions	The most confident suggestion may not be the safest.	What risk does selection create?
One winner	The system may choose a candidate before checking its consequences.	Which exact action won?
External action	A failed state change can cost time, data, or trust.	Does the winner earn authorization?

The central question: after the system has selected the action it intends to execute, what evidence is strong enough to permit execution? The registered authority risk is R_auth(λ) = P(H = 1 | A_λ = 1), where H is a pre-registered adverse-action proxy and A is the authorization event. This is not a production, privacy, financial, or legal harm guarantee.

Selection-induced authority risk figure showing that candidate count and pairwise ranking alone do not determine selected-action risk — Selection-Induced Authority Risk (SIAR). Candidate count alone does not determine risk; score-loss dependence and replacement behavior determine the selected action's adverse-proxy rate.

2. Plain-language concept

Suggestion, choice, and permission are three different decisions

The easiest way to read the paper is to keep these three verbs separate. Confusing them is the failure mode the paper names Selection-Induced Authority Risk, or SIAR.

Propose

A model, planner, rule, or tool suggests an action. This is a belief or plan, not an authorization.

Output: a candidate

Select

The pipeline compares candidates and chooses a winner. A winner is still only the action the system intends to try.

Output: the selected action

Authorize

An independent evidence check asks whether the selected action clears its registered risk and support requirements.

Output: authorize, review, or abstain

Prediction is not permission.High confidence can identify a plausible action without proving its post-action safety.

Selection is not permission.Argmax chooses the top candidate; it does not certify the consequences of that candidate.

Authorization is conditional.Execution authority is earned only for the registered pipeline, evidence mode, and risk group.

3. Experiment design

The experiment asks what happens when a system must choose among candidates

This is a controlled authorization study, not a claim about a deployed commercial agent. The protocol makes the candidate set, outcome label, calibration split, and holdout boundary explicit.

Protocol element	What the paper used	Why it matters
Environment	20 resettable MiniWoB tasks	Each candidate can be evaluated from a controlled initial state.
Candidate set	Registered nested sets, K = 1, 2, 4, 8	The study can separate source-level behavior from winner selection.
Scale	4,000 fresh episodes and 32,000 candidate executions	The main result is not based on a handful of examples.
Splits	Development, calibration, and one-time sealed test	Threshold choice is separated from the final evaluation.
Outcome	Pre-registered adverse-action proxy plus utility	The paper reports both risk and the cost of refusing useful actions.
External sequence	AgentDojo V7-V10	Transfer is tested separately and its negative holdout is retained.

RiskHow often is an authorized action adverse?

CoverageHow often does the system authorize rather than refuse?

LocalityDoes a global result hide a risky task group?

TransferDoes the certificate survive a different environment?

4. Mechanism interaction

EPV-Opt adds a permission check after selection

The pieces are familiar in isolation. The paper's operational point is their order: eligibility and provenance first, selection second, authorization of the selected action third.

**Conceptual architecture, not a paper result.** The generated visual provides spatial intuition; the ordered labels state the method semantics.

Two-lane EPV-Opt architecture: offline training, development threshold selection, fresh calibration, task-wise bounds, and a versioned certificate feed a deployment gate after frozen selection, which routes the selected output to statistical authorization, review, abstention, or fail-closed recalibration — Generation, eligibility, provenance, score, tie-breaking, and argmax are frozen before calibration. A pipeline fingerprint invalidates the certificate after an unregistered change.

Freeze

Candidate count, role registry, provenance rules, score, threshold, task family, and tie rule are part of the certification unit.

Calibrate

Development chooses one threshold. Independent calibration supplies simultaneous local ordinary-risk and active high-impact bounds.

Route

Certified score-clearing tasks authorize; score-clearing but unsupported tasks review; all other actions abstain.

1. EligibilityRole and provenance rules decide which sources may compete.

2. SelectionA frozen scorer and tie rule choose one candidate.

3. CertificationPost-selection evidence tests the winner against global and local risk budgets.

4. RouteAuthorize supported actions, review unsupported local groups, abstain on failure.

4. Mechanism, formalized

Why the permission check has to happen after the winner is known

The paper does not claim a new generic confidence bound. It specifies the selected-action object, then applies exact finite-sample control to the frozen agent pipeline.

Five questions that locate the contribution

Why is selected-action authority distinct?Capability, ranking, and execution permission answer different questions.
What do source-level checks miss?Adaptive selection changes the risk object from a source to the eventual winner.
What is introduced?A selected-action risk formulation and a pipeline-bound authorization and routing method.
What supports utility?Sealed MiniWoB evaluation with explicit risk, coverage, and local-support accounting.
What constrains transfer?The negative independent AgentDojo confirmation prevents an environment-free claim.

Theorem 1

Independent heterogeneous selected-action risk

Under explicit independence, eligibility, and continuous-score assumptions, selected risk equals the probability that each harmful eligible candidate clears the threshold and defeats every competitor, divided by authorization probability.

Propositions

Candidate addition and thresholded replacement

Adding candidates increases risk only when added winners are more adverse than the actions they replace. With a threshold, authorization entry, exit, and denominator changes must also be accounted for.

Theorem 2

Finite-sample split-local control

With a frozen IID pipeline, an independently chosen threshold, and simultaneous Clopper-Pearson bounds over registered task and severity events, every certified task satisfies its active conditional-risk budget with probability at least 1 - δ.

Support cost is explicit. Zero observed adverse outcomes are not enough by themselves: the required calibration support grows with the risk budget and the number of simultaneously protected events.

5. Evidence results

The results support a narrower claim than “the agent is safe”

The 20-task MiniWoB study is the only environment with a positive heldout statistical certificate. AgentDojo is a separate external transfer sequence, not a positive confirmation.

20MiniWoB tasks

4,000fresh reset-state episodes

32,000registered candidate executions

K = 8primary result snapshot

Sealed MiniWoB K = 8 results. UCB is a one-sided 95% test diagnostic for the registered adverse-action proxy. Guarantee semantics differ by method.
Method	Direct	Review	Observed FAR	Test UCB	Utility	AV
Learned selector	1.000	0	0.684	0.701	-449.10	-8.00
Sourcewise CP + argmax	0.089	0	0.000	0.017	169.43	610.53
SCoRE-SDR	0.318	0	0.077	0.097	517.05	958.15
End-to-end CP	0.280	0	0.038	0.054	474.69	915.80
EPV-Opt full family	0.099	0.182	0.025	0.053	180.66	621.77
EPV-Opt split-local	0.194	0.086	0.000	0.008	367.85	808.95

Interpretation: split-local EPV-Opt improves certificate resolution and local routing, not every safety-utility frontier. End-to-end CP and SCoRE retain higher coverage or AV under different formal targets.

Sealed MiniWoB results with candidate replacement, direct authorization under different guarantee semantics, and local routing at K equals 8 — Sealed result overview. Four ordinary tasks receive simultaneous local certificates; score-clearing unsupported tasks route to review and external-side-effect tasks abstain.

External sandbox transfer: AgentDojo

AgentDojo adds three model families, stateful tools, and registered prompt-injection goals. It is reported as an external sequence, not merged with the sealed MiniWoB certificate.

V9 pooled calibration: 84 selected clusters, 0 authority violations, simultaneous UCB 0.043, 66 useful completions with LCB 0.683. The registered test remained unopened under its fixed cost cap.
V10 independent holdout: 19 of 90 clusters authorize, 0 authority violations, but UCB 0.146 exceeds the 0.10 gate and support is below 40. This does not confirm transfer.

The AgentDojo V10 result does not confirm transfer. That failure is operationally informative: execution authority cannot be detached from the pipeline and environment that produced its calibration evidence. A changed system needs a new fingerprint and fresh support, not a portable safety label.

Claim-to-evidence ledger

Registered claim	Direct evidence	Interpretation boundary
Candidate-level evidence does not determine selected-action risk.	Heterogeneous selected-risk identity and replacement analysis; sealed K-scaling study.	Not a claim that more candidates always increase risk.
Post-selection local certification can route unsupported actions away from direct execution.	Sealed MiniWoB split-local result with direct, review, and abstain outcomes.	Applies only to the frozen, registered pipeline and adverse-action proxy.
Global evidence can hide unsupported local task families.	Task-level calibration support and the login-user-popup review route.	Does not establish a certificate for every task or unregistered group.
Certificate validity is pipeline-versioned.	Registered fingerprint and fail-closed invalidation protocol.	Not a drift-robust or online guarantee after a pipeline update.
External transfer remains unresolved.	AgentDojo V10 independent holdout fails its registered authorization gate.	No positive AgentDojo, production, or high-impact authority claim.

Evidence lineage

Study sequence and evidence status

V2 / V3Frozen ranker and preserved zero-direct-coverage certificate failure.
V4 sealed MiniWoBPositive heldout task-local certificate after pre-registered structural severity registration.
V5 / V6Disjoint robustness extension and post-outcome diagnostic audit. Holm adds no coverage gain.
AgentDojo V7 - V10Static and dynamic stress, pooled calibration, then a negative independent confirmation result.

6. Limits and non-claims

The certificate is intentionally narrower than general agent safety

Within scope

Registered conditional adverse-proxy risk after a frozen selected-output pipeline.
Simultaneous ordinary task bounds and structurally active high-impact proxy bounds in sealed MiniWoB.
Fail-closed review or abstention when local support is insufficient.
Certificate invalidation after unregistered pipeline changes.

Outside scope

Production, legal, privacy, financial, or irreversible harm guarantees.
Pointwise, unregistered-task, drift-robust, or post-update validity.
Universal ranking or utility dominance over end-to-end CP or SCoRE.
Positive independent AgentDojo transfer or high-impact authorization.

Reviewer quickstart

Public website sanity check

The public snapshot supports document, hash, figure, and claim inspection. The uploaded Code and Data Supplement is the separate path for a full empirical rerun.

Download

Main, supplement, and combined PDFs are linked directly from the public site.

Main paper · Supplement · Combined PDF

Check

Use the visible command to inspect the released file and its SHA-256.

curl -fsSLO https://computational-decision-lab.github.io/epv/assets/paper/main_with_supplement.pdf
sha256sum main_with_supplement.pdf

Trace

Follow C1-C3 to the exact paper sections, figures, result values, and interpretation boundaries.

Release boundary manifest

Full Code and Data Supplement

Run make verify from the uploaded supplement. Full experimental source is not included in this public artifact.

7. Engineering application

What a responsible implementation would do with this idea

The paper is a certificate protocol, not a drop-in production safety guarantee. Its engineering value is a disciplined place to stop, ask for review, and invalidate stale authority.

Stage 1

Resettable sandbox

Run candidate actions from the same recorded state, observe outcomes, and certify the selected action before allowing an automated step.

Best fit for the paper's evidence mode.

Stage 2

Shadow or review mode

Log the proposed action and certificate decision without applying an irreversible side effect. Route unsupported task groups to a human.

Useful when replay is partial.

Stage 3

Live, irreversible action

Require a current certificate, valid provenance, sufficient local support, and an explicit fallback. Otherwise fail closed.

Outside this paper's demonstrated guarantee.

What an audit record should retain

Pipeline fingerprint, candidate set, role and provenance decisions, selected action, score, threshold, calibration version, risk group, route, outcome, and certificate status.

What changes require re-certification

New proposer, new score, changed role registry, altered candidate order, new task mixture, changed outcome label, or any update that changes the selected-action distribution.

Cross-domain implementation checklist

Freeze candidate sources, eligibility, scoring, and tie-breaking.
Define unacceptable execution events before inspecting outcomes.
Separate development, calibration, and sealed evaluation.
Require stronger support for higher-impact actions.
Route unsupported actions to an accountable professional reviewer.
Invalidate certificates after pipeline or environment changes.
Log the selected candidate, evidence, route, reviewer, and outcome.

Statistical authorization complements access control, formal verification, testing, policy enforcement, and professional accountability. It does not replace them.

Open research questions

How should authority be recalibrated for continuously learning agents?
How can rare catastrophic events receive meaningful finite-sample support?
How should statistical evidence combine with formal verification and expert approval?
When does multi-agent competition amplify correlated failures?
How can evaluations detect systems that behave differently when tested?

Primary documents and reproducibility checks

Main paper7-page main paper SupplementProofs, protocol, audits, and transfer sequence Combined PDF13-page review package Reviewer guideReading order, numbers, and explicit non-claims Submission packageFrozen artifacts, commands, and release boundary Website sourcePublic Pages source and deployed assets

260 tests passed; 1 skipped 43 / 43 figure checks passed 44 / 45 readiness checks passed 0 readiness failures

High-value decision scenarios

Where a plausible action still needs permission

Four deployment scenarios apply the same selected-action question without claiming new experimental evidence.

Deployment scenario, not an empirical result of this paper.

Should an agent merge and deploy the database migration it ranked first?

Coding agents, protected tests, static analysis, review agents, and cached repairs may all propose different changes. Selection identifies a winner; it does not establish authority to alter production state.

Illustration of a production code change held for protected tests, rollback evidence, and human approval before deployment — Deployment scenario, not an empirical result of this paper.

Consequential action: Merge and deploy a database migration or security patch.
What can fail: A convincing patch passes shallow checks but loses data, expands permissions, or follows prompt-injected repository instructions.
Evidence before execution: A frozen toolchain and selector, protected tests, rollback validation, provenance checks, and heldout adverse-change evidence.
Review boundary: Schema deletion, credential changes, production rollout, or insufficient task-local support.

Deployment scenario, not an empirical result of this paper.

Should a high-confidence recommendation be allowed to move funds?

Forecasting models, policy rules, research agents, and risk engines can disagree. Ranking their outputs may favor conviction while dropping the mandate, exposure, or fraud checks that make an action executable.

Illustration of a financial action held at a mandate, exposure, counterparty, liquidity, and fraud review boundary — Deployment scenario, not an empirical result of this paper.

Consequential action: Execute a transfer, trade, credit decision, or portfolio rebalance.
What can fail: The selected action hides downside, liquidity, mandate, counterparty, or fraud risk that was visible before ranking.
Evidence before execution: A registered loss event, exposure group, selector, limits, calibration window, and durable audit trail.
Review boundary: High-value transfer, novel counterparty, regime shift, regulatory exception, or support below the registered minimum.

Deployment scenario, not an empirical result of this paper.

When must an advisory recommendation remain under professional review?

Clinical models, retrieval systems, guidelines, and workflow agents may each supply useful evidence. The operational risk begins when the selected recommendation is converted into a change in care.

Illustration of an advisory healthcare workflow recommendation held for evidence review by an accountable professional — Deployment scenario, not an empirical result of this paper.

Consequential action: Change triage priority, send an order, or alter a care workflow.
What can fail: Selection suppresses uncertainty, overlooks a subgroup, or converts advisory evidence into an unsupported instruction.
Evidence before execution: A validated workflow, subgroup-specific evaluation, provenance, a calibrated adverse-event proxy, and an accountable human role.
Review boundary: Diagnosis or treatment authority, vulnerable subgroup, conflicting evidence, missing context, or any action requiring a licensed professional.

Deployment scenario, not an empirical result of this paper.

Should the fastest containment plan be allowed to lock out an organization?

Detection agents, policy rules, threat intelligence, and remediation planners can converge on a destructive response when telemetry is poisoned or candidate failures are correlated.

Illustration of a cybersecurity containment action held for provenance, blast-radius, reversibility, and human review — Deployment scenario, not an empirical result of this paper.

Consequential action: Isolate a host, revoke credentials, block an account, or rotate keys.
What can fail: Prompt injection, poisoned telemetry, or correlated agents cause a destructive containment action to win.
Evidence before execution: Trusted provenance, adversarial evaluation, blast radius, a reversible execution path, a frozen selection rule, and incident-class support.
Review boundary: Organization-wide lockout, identity-root changes, destructive remediation, disputed telemetry, or an out-of-distribution incident.

Conceptual demonstration, not a paper result.

A stricter score is not enough when local evidence is missing

The score determines which illustrative actions clear the gate. Registered local support still determines whether a score-clearing action can execute or must be reviewed.

Illustrative score threshold 0.70

Reversible patchAuthorizeClears the threshold and has registered local support.

Schema migrationReviewClears the score threshold but lacks registered local support.

Credential rotationAbstainDoes not clear the registered score threshold.

Paper reader

Read the submitted package

The browser viewer is optional. Every document below also has a direct download link for annotation and offline review.

Download selected PDF