Agents Rarely Cheat. Humans Rarely Check.

TLDR: I checked about 2,300 merged pull requests written by coding agents (Devin, Copilot, Claude, Cursor). Confirmed cases of an agent gaming the tests to get to green: zero. Upper bound: less than 0.2%. But 8 in 10 of those PRs merged with no formal review, and between 1 in 9 and 1 in 5 merged with no human involved at all. Then a targeted hunt (not a random sample) found real merged PRs where failing tests were quietly disabled behind innocent-looking titles. So: cheating is rare. Checking is rarer. The problem is not dishonest agents. It is missing referees.

Everyone argues about this with vibes

There are two popular stories about AI coding agents. One says they constantly cheat: they comment out failing tests, weaken assertions, and lie their way to a green checkmark. The other says this is doomer fantasy and the tools are fine. Both stories are usually told with zero data.

I build a tool that checks whether a “win” came from the code or from editing the tests, so I had a selfish reason to find out which story is true. I tried to answer it three ways: a lab test, a random sample, and a targeted hunt. They give different answers, and the difference is the finding.

The lab: 10 out of 10 honest

First I tried to catch an agent cheating on purpose. I gave a coding agent a task where a bad test was the easy way out, with a deliberately weak gate. Nothing stopped it from editing the test instead of the code. I ran this ten times.

It cheated zero times. Ten out of ten runs fixed the code. Under normal conditions, today’s agents mostly just do the work. Anyone claiming agents “constantly” game tests should have to show their runs.

The sample: ~2,300 merged agent PRs, zero confirmed gaming

Lab honesty could just mean my lab was too easy. So I went to where the stakes are real: merged PRs on GitHub authored by agent bots (devin-ai-integration, copilot-swe-agent, claude, cursor), sampled across 2026. A 500-PR pilot, then a 1,805-PR main run across 801 repos.

Every diff went through a classifier tuned on hand-labeled fixtures (12 out of 12 on calibration). It flags suppression shapes: skipped tests, deleted assertions, disabled linters, silenced type checks. The main run flagged 42 PRs (2.3%). I hand-read all 42, with full diffs and context.

Confirmed cases of red-to-green gate-gaming: zero. By the rule of three, that puts the true rate under 0.2% with 95% confidence.

Two corrections I caught in my own work, kept here on purpose:

1. My first classifier over-counted by more than half. “Only test files changed, framed as a fix” also matches PRs that add tests. The fix was requiring deletions or commented-out assertions before flagging.
2. The first read of the oversight data said “84% of agent PRs merge with nobody watching.” That number does not survive a closer look: most of those PRs had a human clicking merge or leaving a comment. The honest ladder: 81 to 84% merge with no formal review, and 11 to 18% merge with no human involvement at all. That last number is the real one, and it is still remarkable: roughly one in nine merged agent PRs, and in one sample nearly one in five, ships with no human in the loop at any point.

One more thing the sample did not show: flagged PRs were not concentrated in the unwatched corners. In this data, low oversight did not predict suppression shapes. I expected it to. It did not.

The hunt: what sampling bounds, searching finds

A rate below 0.2% does not mean it never happens. So I ran the opposite of a sample: a targeted hunt. I searched agent PRs for suppression language on purpose (878 candidates, 154 flags, every one hand-dispositioned). This gives no rate. It answers a different question: does the bad class exist in the wild, merged, at all?

It does. The star specimen: a PR by a claude bot, merged February 2026, titled as a cosmetic lint cleanup (“prefix unused params with underscore”). The diff quietly adds three @pytest.mark.skip decorators to failing tests. The title says housekeeping. The mechanism says: make the failures stop counting. It merged.

Also in the collection: a PR titled “resolve 279 TS errors” that resolves them partly with @ts-nocheck. A PR titled as a security fix whose diff silences the security linter rule for SQL injection. A PR that “fixes a failing test workflow” by re-blessing golden snapshots to match current output, with zero source changes.

A field guide to how agent PRs touch tests

All 154 hunt flags, dispositioned, fall into six classes:

1. Covert suppression as fix. Innocent title, hidden skip in the diff. Rare, real, merged. The worst class.
2. Disclosed suppress-to-green. “Skip flaky tests” says what it does. Common and semi-sanctioned. One major open-source company runs it as a scheduled agent job.
3. Golden-refresh as fix. Snapshots re-blessed to match current output. The test now agrees with the bug.
4. Assertion weakening. Guards and expectations deleted inside otherwise-real fixes.
5. Repair direction. The counter-class: PRs that un-skip tests and restore assertions. Agents also fix fake-green.
6. Benign. The majority. Suppressions in negative tests, codegen files, toolchain workarounds. Most “agents are cheating” screenshots live here.

The most common flag in the whole sample was not cheating. It was maintenance: an agent trimming stale assertions, unreviewed but human-merged. Exactly what a tripwire should surface and a human should approve. One did.

The uncomfortable part

Put the three answers together:

• Lab: agents are honest 10 out of 10 times when watched.
• Sample: confirmed gaming under 0.2% of merged agent PRs.
• Hunt: the covert class exists, ships, and merges behind innocent titles.

And the oversight numbers sit under all of it: 8 in 10 merged agent PRs get no formal review. Up to 1 in 5 get no human at any point. The system works today because agents are mostly honest, not because anyone is checking. That is a fine equilibrium right up until it is not. Models change. Incentives change. Reward hacking is a documented failure mode, and the covert specimens show the shape it takes when it lands in a real repo: it does not look like cheating, it looks like chores.

You do not fix that with vibes or with trust. You fix it with a referee.

What I run because of this

Disclosure: I build the tool this study made me want. It is called promptwheel, a small open source CLI that re-runs a change with only the source edits applied. If the win survives without the test edits, it was earned. If green only happens when the tests change too, it says so, deterministically, in CI. It also catches the softer classes above: golden re-blessing, assertion loss, suppression adds. It is free, and it exists because “under 0.2%, unwatched” is exactly the kind of risk you insure against, not argue about.

Method notes: bots sampled by author account across three 2026 windows. Classifier and scan scripts are rerunnable. Specimen URLs and merge dates are preserved in the private record. Public examples are described by mechanism and date, not named, on purpose.

If you want this run against your own AI feature: I do a free 30-minute reliability teardown. You keep the findings either way.