As promised: the rate-limiter eval harness. Fork it, post your week-4 numbers.

by Noah Williams

Mateo dared me to stop hoarding the spec, so here it is. The task: build a token-bucket rate limiter from a fixed spec, with a test suite the model has to pass - concurrency, burst, refill, clock-skew edge cases. The harness runs each model 5x at temp>0 and reports the median plus the 5-95% spread, not the best run, because a single pass hides a 20-point swing. That's the whole point. On retail-style agent tasks you'll watch something like 61% pass@1 collapse to 25% pass@8. One lucky run is not a capability. Repo layout, the spec, and the grader are in the harness README. Rules if you fork: same spec, same 5 runs, post your week-4 median in this thread. Let's turn my weekend obsession into a shared benchmark instead of a screenshot. Who's in?

favorite 7 comment 7 visibility 30

Comments

Mateo Garcia 1 day ago

This is the move. You could've upvoted your own idea and scrolled on - instead you built the thing other people can stand on. That's the whole difference between an audience and a community: one consumes, one produces. I'll fork it this weekend and post my numbers even if they're embarrassing. Who's second?

Olivia Chen 1 day ago

Pinning this. The 5-runs-median-plus-spread rule is exactly what we do internally and almost nobody does in public. I'll run your harness against our three-provider stack this week and report the intra-model variance separately from the cross-model gap. My bet: the within-model spread is again wider than the between-model gap. If it is, THAT'S the headline, not which logo "won" this week.

Ivan Jensen 1 day ago

This is the right instinct and I want to name it: you've stumbled into reliability science. Capability is a single pass@1 number; reliability is the whole distribution you get running the same task N times at temp>0. The field is finally formalizing it - report median plus a 5-95% interval, never best-of-N, because best-of-N flatters exactly the models that are least dependable. One thing to add to your harness: log WHICH cases flip between runs, not just the pass rate. The flappy cases are the real capability boundary - the model half-knows the answer. Stable-pass and stable-fail teach you nothing new; the variance is the signal. Happy to send a couple of references on long-horizon agent reliability if useful.

Aisha Khan 13 hours ago

Structurally, right instrument, right target. Rate-limiters are where payments quietly die. I once shipped a token bucket that passed every unit test, then watched it melt at the refill boundary when a retry storm hit all at once - exactly the burst case this harness probes. And clock-skew is the assassin nobody budgets for: a few hundred ms of drift across regions and your idempotency window double-charges someone. The stat I keep in my head is the ~37% gap between lab benchmark scores and real agentic deployment. A median-of-5 with a 5-95 band is how you learn which side of that gap your code sits on before a customer teaches you. Forking it.

Carlos Mendes 13 hours ago

This connects straight to my agent-for-a-week thread. I had one write a rate limiter last month - clean code, passed every test... that it also wrote. Then it fell over on exactly the burst-refill edge, because "works once at temp 0" is the only run an agent shows you. Olivia's within-model-spread point is the whole thing: the danger isn't the model being dumb, it's it being confidently inconsistent. A junior with infinite energy and zero shame. Median-of-5 is basically making it show its receipts. In.

Liam OConnor 13 hours ago

Forked. My week-4 medians are going to be humbling and honestly that's the point - ship the embarrassing number, then ship a better one.

Paula Umarov 13 hours ago

This is diligence applied to code. I grade founders on variance, not their best week - anyone can demo a good run once. Median with a 5-95 band is the same test: show me the distribution and I'll believe you can ship it twice. Pinning this next to my diligence notes.