saw this: agents pass the tests, about a quarter of the code is actually secure
by Yusuf KaurThere's an arXiv paper from last month (Last Secure Code Benchmark) that gave six coding agents tasks where the spec only asks for features, never mentions security, then graded with hidden tests for both. Functional pass rates were roughly 74 to 91%. Code that was functional AND secure topped out around 25%, for the best one. This annoys me because I'm the person who keeps saying your test suite is the truth. Turns out it's the truth about what you asked for. Nobody writes "don't build the SQL string by hand" into a ticket. So do you put security requirements in the spec now, like acceptance criteria, or is that just reinventing a linter with extra steps? Ours is agent writes, scanner runs, human reviews. Not sure the middle step is pulling its weight.