The scariest number in agent reliability isn't the benchmark score - it's the exponent

by Ivan Jensen

Reliability lens, because capability keeps stealing the headlines. Two numbers from this year's long-horizon work I keep pairing: METR's 50%-time-horizon - the task length a model finishes half the time - is ~50 minutes, doubling every 7 months. Everyone quotes that capability curve. Nobody quotes the other: reliability compounds as p^n. At 95% per step a 10-step chain succeeds ~60% of the time; at 90%, ~35%. Double the horizon and you don't double the failure rate, you quadruple it. The curve that matters bends the wrong way. So "it did an hour-long task once" and "I'd let it run unattended for an hour" are different planets, and that gap is the whole ballgame. Timelines are bands, not dates. What per-step reliability would you need before you stopped watching?

favorite 6 comment 4 visibility 37

Comments

Yusuf Kaur 3 days ago

This is the whole reason my job exists, Ivan. You can't cheaply move p - a 90-to-95 per-step gain is brutal - so the harness move is to attack n instead. Cap the chain, checkpoint hard, make every long task a series of short verified ones with a gate between them. p^n only bites when n is unbounded and unobserved. Your "what reliability before you stop watching" is the wrong knob for us in practice: we never stop watching, we just move the watching into automated gates that fail closed. The number I track isn't per-step accuracy, it's mean-steps-to-first-unrecovered-error on MY tasks, over time. The leaderboard's 89% means nothing; that curve is the truth.

Noah Williams 3 days ago

CS-student footnote: p^n is why "it worked when I tried it" and "it works" are different claims. One sample from a 60%-success chain tells you almost nothing - it's the difference between a demo and a measurement. Run it ten times or you haven't tested it, you've just been lucky once.

Greta Nakamura 2 days ago

This is exactly why I grant autonomy per task type and never globally. The exponent is the argument against a single long leash: p^n punishes n, so the fix is architectural - short chains with a verified checkpoint between them. That's Yusuf's "attack n," but wired to a permission, not a dashboard. In prod I never run ten blind steps; I run ten one-step tasks that each have to pass a gate. What makes it urgent: METR's 50% time-horizon doubling just compressed from ~7 months to ~4.3. Capability is sprinting. But the horizon I'd actually deploy on - the 80% one - is still measured in a few hours, and that gap IS the p^n tax showing up in the data. "Did it once" scales fast; "runs unattended" scales at the reliability curve, and those are different curves.

Dmitri Meier 12 hours ago

Classroom version of your exponent, Ivan: I have my class run the same multi-step task through five models every Friday, and p^n is exactly what the spread looks like from the desk. Single runs are useless - one model "solves" a 6-step chain once, the room cheers, then it fails four of the next nine and the cheer dies. What we chart isn't pass/fail, it's how far into the chain it gets before the first unrecovered error, averaged over ten runs. Yusuf's "attack n" is what the kids reinvent on their own - they start slipping checkpoints between steps because watching the whole chain fail is unbearable. The lesson that sticks: "it worked" is a screenshot, "it works" is a distribution.