Elwen: let an agent go find what broke.

Something quietly broke in the last year. We got very fast at shipping. A new endpoint, an auth change, a data migration, a tweak to a page layout - the kind of change that used to take a day now takes an agent twenty minutes. But the way we check that change still assumes the old speed: write assertions against fixtures, run them, ship, and hope staging matched reality.

The assertions only cover what you already thought of. That was fine when a human wrote every line and knew where the bodies were buried. It is not fine when the model wrote the diff and neither of you has poked the running thing yet.

There is a different move available now, and it is the one a good manual tester makes: open the live service, try a handful of requests, chase the edge case, notice the endpoint that returns data it shouldn’t. Agents are unreasonably good at exactly this. The problem is that “run Claude against a long prompt” is not infrastructure. Elwen is the thin layer that makes it infrastructure.

Elwen is a single Go binary that turns an agentic CLI - claude, codex, whatever you already run - into a repeatable probe pipeline. You send it an event describing what you just shipped and where it lives. It points an agent at the live target, lets it poke around, and hands you back typed findings with evidence and reproduction steps. No dashboard, no YAML, no cloud lock-in.

The shape of a probe

Every run is one event that walks the same five stages. The interesting part is that only one of them costs you agent time.

The event is deliberately small. You describe the target and the scenario the way you’d brief a person:

{
  "destination": "https://staging.example.com",
  "service": "checkout-api",
  "scenario": "verify checkout rejects expired coupons; look for endpoints that leak data or skip auth",
  "profile": "web",
  "params": { "auth_token": "...", "scope": "checkout only" },
  "budget": { "timeout_sec": 1800, "max_turns": 30 }
}

That scenario field is the whole idea. You are not encoding assertions. You are handing over intent - and a budget so the run can’t wander off forever.

What you get back

Run it against the bundled shop API and the output reads like a tester’s notes, not a stack trace:

=== shopdemo / run_shopdemo  http://127.0.0.1:8080
  ✓ pass  reachability             destination responded with HTTP 200 OK
  ! warn  response-headers         missing baseline response headers  [medium]
  ✓ pass  openapi-spec             loaded spec: 5 paths, 6 declared checks
  ✗ FAIL  expired-coupon-rejected  POST /coupons/validate: field "valid" = true, expected false  [high]
  ✗ FAIL  orders-requires-auth     GET /orders: expected HTTP 401, got 404  [critical]
  ─── agent posted /done  ·  16 results  ·  pass=10 fail=5 warn=1
  trace → logs/shopdemo/run_shopdemo.md

Every line is a typed result, not a blob of stdout. Each carries a status (pass | fail | warn | info), a severity, a one-line summary, and - this is the part that makes it usable - the evidence:

{
  "check": "expired-coupon-rejected",
  "status": "fail",
  "severity": "high",
  "summary": "POST /coupons/validate accepted EXPIRED10; field \"valid\" = true",
  "evidence": { "request": "POST /coupons/validate {\"code\":\"EXPIRED10\"}", "status_code": 200 },
  "remediation": ["reject coupons past expiry before returning valid:true"]
}

That’s the difference between “a test failed” and “here is the exact request that misbehaves, what it returned, and what it should have done.” You can hand a finding straight to whoever wrote the diff - or to the agent that wrote it - and it has everything needed to reproduce.

Why this beats one more test suite

It doesn’t replace your unit tests. It sits next to them and does the thing they structurally can’t.

A fixture can’t tell you the new /orders endpoint returns 404 instead of 401 because it was never wired up. A mock can’t notice that an endpoint you forgot about answers unauthenticated. Those are the failures that survive a green build and surface in production - and they’re precisely what an agent poking a real URL trips over in the first minute.

Not just APIs

I keep reaching for API examples because they demo cleanly and the repo ships an OpenAPI target you can watch fail and recover flag by flag. But nothing about Elwen is API-shaped. The event carries a profile - web, api, security, or your own - and a free-text scenario. Point it at a rendered page and ask “did the new pricing table actually render and are the CTAs clickable.” Point it at a service after a data migration and ask “spot-check that migrated records still resolve.” The harness doesn’t care; it just gives an agent a live target, a goal, and a budget, and collects what comes back.

The loop it closes

Here’s where it lands in a real pipeline. Nothing exotic - it hooks into the deploy you already have:

Because it reads events off stdin, the producer can be anything: a CI step, a queue message, a cron, a webhook. Elwen deliberately ships no queue client - it only cares that JSON reaches its input. That keeps it a component you drop into whatever you already run, not a platform you migrate onto.

The recurring findings point at a second move, too. Anything the agent keeps rediscovering - the same header missing, the same contract violated - is a candidate to promote into the deterministic meta layer, where the next run catches it in milliseconds instead of spending a turn on it. That’s the intended arc: the expensive, exploratory layer teaches the cheap, fast one, so over time the agent is freed up to look only for what’s genuinely new. Today you make that promotion by hand; having Elwen suggest the checks itself is on the roadmap.

That’s the whole pitch. Not a smarter test framework - a margin layer that does the poking a careful human would do on every deploy, at agent speed, and writes down exactly what it found.

The code is open source and MIT licensed: github.com/msoedov/elwen. make demo runs the whole thing in about ten seconds with no API key.