AI vulnerability patching has a grading problem
Researchers at Northwestern University tested AI coding agents on 112 real security bugs. They found that a patch which stops a crash often fails the checks the project’s maintainers wrote for the fix. With Claude Opus 4.7, third of the patches that made the crash go away still failed those maintainer tests.
If you’re thinking about letting an AI tool patch vulnerabilities for you, this is the number to look at. Many security patching benchmarks grade a fix on one question: does the proof-of-concept input, the sample that triggers the bug, still crash the program? A patch can pass by blocking the path to the crash and leaving the flaw where it was. The authors call that kind of fix “strictly worse than no fix at all,” since it tells you the problem is solved when it isn’t.
Two patches, one right answer
The researchers built their dataset from bugs in 84 open-source projects written in C/C++, Go and Rust. For each bug, they pulled the tests the original developers added alongside the real fix. Then they ran the AI patches against those tests.
Take a bug in PHP. One AI-style patch fixes the logic the way PHP’s maintainers did. Another just throws an error whenever the inputs look unusual, which stops the crash but also breaks calls PHP is supposed to handle. Both pass PHP’s existing tests. Only the test the maintainers wrote for the fix tells them apart.
The team also tried three generations of Claude models. The newest stopped the crash about 90% of the time, but it passed the maintainers’ tests only about 61% of the time. So even the best model they tested failed those tests on roughly four in ten bugs.
The agents found the answers
The team also caught agents pulling answers out of the test setup:
- Web search: with web search left on, one Claude Code agent fetched the upstream fix commit within its first four tool calls. It later reported that its patch followed the same approach and cited the page it had fetched.
- Git history: another agent was given the project’s full git history. It searched for an earlier related fix and reproduced it almost line for line.
- Answer key: in an earlier version of the harness, an agent read a configuration file that pointed to the ground-truth patch. Its edits matched the upstream fix exactly.
The authors point out that searching the git history was a reasonable move, the same thing a human developer would do. The benchmark had left the answer in reach.
The setup also caused failures that weren’t the AI’s fault. A hidden background check in Claude Code made commands stall, which looked like the model struggling. In another case, an agent wrote a correct fix plus its own test, and that test clashed with the maintainers’ test file. The benchmark scored the correct fix as a failure.
The team reviewed 13 benchmarks and datasets used to evaluate AI patching. Each of the five built for AI agents left at least one of the nine pitfalls unaddressed.
What to ask a vendor
If you’re sizing up an AI remediation tool, here are four questions to ask:
- Did the score check only that the crash stopped, or did it also run tests the maintainers wrote?
- Did the agent have internet access or the project’s git history?
- What did the prompt tell it about building and testing the code?
- How were timeouts and pipeline errors counted?
That third question matters more than you’d think. Just telling the AI how to build and test the code bumped its success rate on the C and C++ bugs by 20 points, without the AI getting any smarter.
And on a leaderboard that only checks whether the crash stops, every one of those patches that missed what the maintainers intended would have counted as a win.
Download: The IT leader’s guide to AI code sprawl
