If you run an AI coding loop — Claude Code’s /loop, a Ralph while-loop, an agent grinding on a backlog — you’ve felt the failure mode even if you haven’t named it: the loop optimizes what it can measure and confidently degrades what it can’t. A prompt tweak “passes the tests” and quietly drops answer quality. A refactor lands and the p95 creeps up. The model says done. Nobody had a number that said better, so nothing caught it.
That gap is the whole reason PromptWheel exists.
The pivot: from orchestrator to outcome gate
PromptWheel started as a batch-improvement orchestrator. In 2026 the base tools absorbed that entire layer — Claude Code shipped Workflows, subagents, hooks, and /loop; Cursor shipped background agents. Building orchestration as a product became building next quarter’s free feature.
But underneath the orchestration, one problem stayed unsolved and unowned: nobody verifies that a change actually improved a real metric without regressing another, and feeds that back as a reward. Outcome data rots in dashboards, disconnected from the change that caused it. So PromptWheel became the one thing the platform didn’t commoditize — the trustworthy per-turn reward signal — and shed everything else.
What it does now
For any change (base → head), PromptWheel measures each metric you care about before and after, in throwaway git worktrees (it never touches your working tree), then emits a verdict:
PromptWheel a1b2c3d → e4f5g6h (×5)
▲ eval_pass_rate 0.58 → 0.92 (+0.34, improved) [guard✓, high]
▼ cost_per_run_usd 0.0014 → 0.0007 (-0.0007, improved) [guard✓, high]
▲ p95_ms 210 → 214 (+4, inconclusive) [info, low]
VERDICT: PASS
A metric is anything a shell command can print — a test pass-rate, an eval score, $/run, latency, bundle size, coverage. One tool, any number.
The part that makes it trustworthy: it won’t lie on noise
Most “did it improve?” checks are worthless because the number jitters between runs. PromptWheel refuses to pretend otherwise. With --repeat N it samples each metric N times, takes the median, and measures the noise band. A delta that sits inside the jitter is reported inconclusive and never fails a guard — no flaky CI, no false alarms. It only calls a change a regression when the regression clears the noise. That single discipline — never assert what you didn’t measure — is the credibility feature, and it’s the difference between a signal and a coin flip.
It’s the signal, not the driver
PromptWheel doesn’t run your loop; it tells the loop whether each turn earned its keep. The improve command makes that a one-liner — run any agent, gate the result, keep the change only if a metric improved, else revert:
# converge: keep spinning while each turn improves something; stop on plateau or regression
while promptwheel improve --attempt "claude -p 'speed up the hot path'"; do :; done
Exit 0 = kept a real win, 1 = guarded regression (reverted), 3 = plateau (reverted). Any driver converges on that contract without parsing anything. And every gated run appends to a per-repo record, so promptwheel insights can tell you which change-types actually move which metrics — the loop’s memory.
Try it
npm install -g promptwheel
promptwheel init # detect your stack, write a starter config
promptwheel run --working # gate your uncommitted changes
Or bring it into Claude Code as a plugin:
/plugin marketplace add promptwheel-ai/promptwheel
/plugin install promptwheel@promptwheel-ai
It’s MIT, zero-dependency, one file. The repo is github.com/promptwheel-ai/promptwheel.
If your team is shipping an AI feature and can’t tell improvement from churn — or your bill keeps creeping — that’s exactly the problem I close in a fixed two-week AI Reliability Sprint: an eval suite, cost guardrails, and a regression gate committed to your repo, built on PromptWheel. It starts with a free 30-minute teardown of where your feature is silently regressing. Grab one.