Verify the delta, not the state.

The eval dashboard says your flow-builder agent passed again last night. Twenty runs, twenty green. Then a customer asks why none of their flows exist, you open the staging workspace, and the truth is ugly: the agent’s create call has been returning 401 since an API key rotated two weeks ago. Every one of those green runs proved nothing. The verifier checked flow_count >= 1 at the end of each run, and the workspace it ran in has held a handful of flows since March. The assertion was true before the agent spent a single token.

This failure has a name. The Verification Design pattern catalog calls the fix Delta, also known as baseline assertion or relative state verification. The whole pattern fits in one sentence:

It sounds almost too small to deserve a name. Then you audit your own agent evals and count how many of them check an absolute value against an environment that was not empty when the run started.

The environment already contains the answer

Agents rarely act in clean rooms. They operate in staging databases that nobody has truncated since the demo, CI runners with warm caches, shared workspaces where three teammates and a cron job are also writing, and live cloud accounts with months of accumulated state. Verification Design calls the residue ambient state, and any absolute assertion made on top of it is contaminated by it. total_flows >= 5 is not a claim about your agent. It is a claim about the union of your agent, everyone who touched the environment before it, and whatever the seed script left behind.

The trap closes quietly because nothing looks wrong. The check runs, the number satisfies the threshold, the report renders green. Play with the figure below: the workspace starts with three flows that some earlier run left behind, and the naive verifier asserts count >= 1. Toggle the agent’s tool call between succeeding and failing silently, and watch what the verdict does.

The verdict does not move. That is the entire problem. A verifier that returns the same answer whether or not the agent acted is not a verifier, it is a mood light. And an agent is the worst possible consumer of a mood light, because a model will happily rationalize a coincidental pass as evidence that its tool call worked. It never learns the call has been broken for two weeks, so it never retries, never escalates, never mentions it. The green report actively hides the outage.

Baseline, act, measure, diff

The pattern replaces the absolute read with a bracket around the agent’s action. You capture the metric immediately before handing control to the agent, let the agent run, capture the same metric immediately after, and assert on the difference. Five moving parts, all boring on purpose:

In code, the antipattern and the pattern differ by a dozen lines. The naive check reads the world once and hopes:

def verify_flow_created_naive(get_flow_count):
    observed = get_flow_count()
    return {"passed": observed >= 1, "observed": observed}

The delta version brackets the run. Adapted from the pattern’s reference implementation:

class DeltaVerifier:
    def __init__(self, metric, fetch_state, expected_delta):
        self.metric = metric
        self.fetch_state = fetch_state
        self.expected_delta = expected_delta
        self.pre_state = None

    def capture_baseline(self):
        """Orchestrator calls this before the agent acts."""
        self.pre_state = self.fetch_state()

    def verify_delta(self):
        """Verifier calls this after the agent acts."""
        if self.pre_state is None:
            raise ValueError("baseline never captured; cannot verify a delta")
        post_state = self.fetch_state()
        actual = post_state - self.pre_state
        return {
            "check": f"{self.metric}_delta",
            "passed": actual == self.expected_delta,
            "pre_state": self.pre_state,
            "post_state": post_state,
            "expected_delta": self.expected_delta,
            "actual_delta": actual,
        }

Run it against the contaminated workspace from the opening scene: baseline reads 3, the agent creates its one flow, the post-hook reads 4, and the check passes because the delta is exactly +1. Now break the agent’s tool call and run it again: baseline 3, post-state 3, delta 0, and the run fails loudly, which is what you wanted two weeks ago.

What a green delta actually proves

Precision matters here, because the pattern is often oversold. A passing delta check proves that the metric changed by the expected amount during this run’s verification window: the span between the pre-hook and the post-hook. That is a much stronger claim than an absolute read, and still a weaker claim than “my agent caused it.” The report should carry everything a reader needs to audit the window: the named metric, the baseline, the post-state, the expected delta, and the actual delta. If your verifier emits only passed: true, you have thrown away the evidence.

The window is also where the pattern quietly rots. Capture the baseline at system boot instead of immediately before the step you are verifying, and every ambient write that lands in between gets billed to your agent. The figure below puts the pre-hook on a slider. The agent contributes exactly one flow, a cron job and a teammate contribute one each at fixed times, and the only thing you control is when the baseline is captured.

Where it breaks

The stale baseline is one failure mode. The other is concurrency. If another actor writes to the same metric inside your window, the delta reads +2 when you expected +1, and your check fails even though the agent did its job. Worse, if your expected condition is a lazy delta >= 1, a neighbor’s write can make the check pass while your agent’s call failed, and you are back to the mood light with extra steps.

The pattern catalog’s answer is to pair Delta with a causal tag: the agent stamps every artifact it creates with the run’s ID, and the post-hook counts only artifacts carrying that stamp. The metric changes from “flows in the workspace” to “flows in the workspace tagged with run 7f3a,” and the neighbor’s write stops being your problem. The figure below runs both actors inside one window; flip the tag on and watch the same race produce a clean +1.

Scoping the metric this way is what upgrades “the state changed during my window” into something close to causal attribution. Without it, delta verification in a busy environment is a coin flip on your neighbors’ schedules. With it, the two patterns cover each other’s blind spots: the tag proves whose artifact it is, the delta proves when it appeared.

When to reach for it, and when to skip it

Delta earns its keep when the environment is shared, persistent, or arrives with non-trivial pre-existing state, and when the metric you care about already exists before the agent acts. That describes most staging databases, most long-lived workspaces, and nearly every “run the agent against our real sandbox” eval I have seen. If concurrent activity is possible, budget for the causal tag at the same time; retrofitting tags after you have a flaky delta suite is miserable.

Skip it when the environment is genuinely ephemeral: a fresh container per run, a mocked database, a workspace created and destroyed inside the test. There, the absolute assertion is simpler and just as honest, because the pre-state is zero by construction. Skip it too when you cannot read the metric before the agent acts, when only the absolute post-state carries meaning, or when the pre-read itself would disturb a destructive, non-replayable action. A baseline you cannot capture cleanly is worse than no baseline, because it manufactures false confidence.

Test authors solved this before agents existed

None of this is exotic. Aider’s command tests do it today: before dropping a file from a chat session, the test records initial_count = len(coder.abs_fnames), and afterwards asserts len(coder.abs_fnames) == initial_count - 1. The check holds no matter how many files were already in the session, because it is a claim about the change, not the count. The flaky-test literature backs the instinct: Luo et al. (FSE 2014) identify test-order dependency, one test polluting state that another test then reads, among the recurring causes of flaky tests. Ambient state by another name.

What agents change is the stakes. A human developer who sees a suspicious green test gets curious. An agent that sees a green verifier takes it as ground truth, folds “the task succeeded” into its context, and builds on the lie. This is why we treat verification design as part of the agent system, not a QA afterthought: the same discipline that gates our eval suites in CI has to hold inside the agent loop itself, where the model reads its own verifier output and decides what to do next.

So audit your evals this week. Grep for >=, == "completed", and any assertion that reads the environment exactly once. Each one is a place where a rotated key, a failed deploy, or a silent tool regression can hide behind a green light indefinitely. The fix costs a dozen lines and two extra reads. Assert on the change, not the state, and a pass starts meaning what everyone already assumes it means: this run did it.

Credit where due: Delta is one pattern in the Verification Design catalog, which names and documents the verification moves agent builders keep reinventing. Worth a full read.