Enterprise AI is entering an evaluation gap: Agents are gaining autonomy faster than companies can verify them

Half of enterprises have deployed an AI agent or LLM feature that passed internal evaluations and yet still caused a customer-facing failure — one in four more than once — according to the June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees.

The sample is self-selected rather than a probability sample, so the findings should be read as directional, not precise.

But enterprises are not responding by slowing automation: 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months. Only 5% say they fully trust the automated evaluations that would make those release decisions.

That mismatch is the evaluation gap: the autonomy ceiling is rising faster than the assurance beneath it. 

It also fits a broader thesis that will be explored at VB Transform 2026: enterprises ship agents first, while the control layers around identity, evaluation, cost, context and orchestration are arriving later. The next year will be a retrofit cycle, with buyers shifting budget toward the systems that make agentic deployments governable and dependable.

Why a passing evaluation is not a working agent

Traditional software testing usually asks whether a defined input produces an expected output. Agent testing is harder because the system may choose its own sequence of steps, call tools, retrieve data, alter state and respond differently from one run to the next.

An agent can make several individually plausible decisions and still reach the wrong result. It may retrieve the correct account but update the wrong field. It may draft a valid refund request but send it without approval. It may call five tools successfully before a sixth step leaks sensitive information or leaves a workflow incomplete.

The survey shows enterprises already recognize this limitation. The most common reason for distrusting automated evaluation is poor alignment with real-world outcomes, cited by 29% of respondents. Bias or inconsistency follows at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%.

That hierarchy matters. Enterprises are saying the score often does not predict what happens when a customer, employee or business process encounters the agent in production — not that automated scoring is too slow or expensive.

NIST makes a similar point in its Generative AI Profile: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context and operating conditions. Its guidance calls for field testing, post-deployment monitoring and clear processes for escalating failures.

Capability is not consistency

A single successful run proves that an agent can complete a task. It does not prove that it will complete the task reliably.

Anthropic’s guidance on agent evaluation distinguishes between measuring whether a system succeeds at least once across repeated attempts and whether it succeeds every time. That distinction is essential for customer-facing or operational workflows. A model that occasionally produces an excellent answer may still be unacceptable if the same task fails unpredictably on the next attempt.

Enterprise teams should therefore treat repeatability as a first-class metric. That means running the same scenario multiple times, varying phrasing and context, testing tool failures, and measuring whether the final business outcome remains correct even when the route changes.

The evaluation set also has to evolve. Every production incident should become a permanent regression test. Customer escalations, failed tool calls, incorrect approvals and data-handling mistakes should feed back into the pre-deployment suite rather than remaining isolated support cases.

Autonomy should expand by risk, not by ambition

The survey does not imply that every agent action should require a person. Human review cannot scale across millions of low-consequence decisions.

But zero-human operation should be earned by demonstrated reliability and bounded by the consequences of failure.

Low-risk actions such as drafting internal summaries or categorizing documents can tolerate broader autonomy. Financial transactions, customer communications, code deployment, access-control changes and data deletion need stricter thresholds, repeated consistency tests, policy checks, rollback mechanisms and clear human escalation paths.

The risk isn't evenly distributed by company size, either. Larger enterprises — those with 2,500 or more employees — are moving toward zero-human deployment fastest, at 70% versus 64% for smaller companies, and they're also shipping more agents that go on to fail a customer, at 54% versus 48%. 

That is the warning for enterprise leaders. Removing the human from the loop does not remove uncertainty. Without stronger assurance, it converts uncertainty into an automated production decision.

The market will keep pushing toward greater autonomy because the economic incentive is real. The organizations best positioned won't be those that remove people fastest — they'll be the ones that treat repeatability and regression testing as seriously as deployment speed.