We checked 317 merged AI-agent PRs against the tickets they closed

Coding agents now open a lot of pull requests that say Fixes #123. We wanted to know how often the ticket is actually done when that PR merges. So we took 317 public pull requests from 317 different organisations. Each one was written by a coding agent (Claude Code, Devin, Copilot's coding agent or Cursor), was merged, and closed at least one issue. We checked every requirement in each issue it closed against the diff.

1 in 7left at least one requirement missed or only partly done (15%)

6%skipped a requirement entirely

1 in 4when the ticket was edited after the agent started (6 of 23)

Most agent PRs do what the ticket asks. The ones that don't are hard to spot in review, because the reviewer reads the diff, not the ticket. The rate nearly doubles when the ticket moves mid-work. That's a small sample, but it's the case we built Lians for.

What gets missed

Of the 70 requirements that were missed or only partly done:

Core behaviour49%

Tests21%

Error / edge cases10%

Docs / changelog10%

UI6%

CI / release4%

What we kept seeing

The last line of the ticket. Acceptance criteria at the bottom, such as "check the other files for the same pattern" or "add a test for the empty case", drop out more often than the headline change.
The description says it, the diff doesn't. "Tests pass", "kept the manifest in step". The claim is in the PR body, and nothing in the diff backs it up.
Yesterday's spec. When a requirement changes after the agent starts, the PR often builds the version it started from.

How we checked, including what we got wrong

Each PR was checked by Lians check. Claude reads the issue the PR closes and its discussion, splits it into requirements, and marks each one met, partial, missed or "can't tell" from the diff.

Our first pass was wrong, and it's worth saying how. It graded every issue a PR so much as mentioned, including "#123 stays separate", and it ignored scope changes in the issue's comments. That run said 44% of PRs missed something. Hand-checking a sample showed most of those were false alarms. We fixed both problems and re-ran everything. On the hand-checked findings, about 3 in 4 of the new run's "missed" or "partial" calls held up. So read 15% as an upper bound, and the true rate as closer to 1 in 9.

A further 323 requirements came back "can't tell", usually because the evidence was a CI run, a sibling PR or code outside the diff. They aren't counted as gaps. The sample is public repositories found through GitHub search, so it leans toward smaller teams and open-source companies. We aren't naming any repository here.

What to do about it

When you review an agent's PR, read the ticket as well as the diff, and look hardest at PRs whose ticket changed after the work started. If you'd rather not do that by hand, Lians check does it on every PR. It runs in your own GitHub Actions, posts one comment listing each requirement, and re-checks the open PRs linked to a ticket when someone edits it.