I ran the same coding task through 5 AI models every week for a month

by Noah Williams

Side project turned obsession: same non-trivial coding task (build a rate limiter with tests, from a fixed spec), five different models, every Friday, four weeks. Results: the top two swapped places twice, the "best" model failed a run the cheapest one aced, and week-to-week variance within the SAME model was bigger than the gap between models. My takeaway as a student: stop arguing about which model is smartest and start building evals for YOUR task. The leaderboard is a vibe; your test suite is the truth. Happy to share the spec if anyone wants to reproduce this.

favorite 44 comment 9 visibility 750

Comments

Maya Patel 1 month ago

This mirrors what I see in AI search: rankings inside one model move week to week even when nothing changed on the site. Everyone hunting a stable "algorithm" is chasing a moving average. Build for the distribution, not the snapshot.

Olivia Chen 1 month ago

This matches my eval work exactly. We run a 200-case suite nightly across three providers, and the intra-model variance is why we stopped hot-swapping "the best" model every month. Pin, eval, migrate deliberately. Publish the spec — I'll run it against our stack.

Priya Nair 1 month ago

Week-to-week variance within one model is the most under-reported fact in applied ML right now. We version-pin like it's 2015 dependency management again. Publish the spec — this deserves replication.

Paula Umarov 1 month ago

What changed my mind recently was watching agents handle 5-step tasks without supervision. Two years ago that was science fiction.

Mateo Garcia 1 month ago

This is exactly the kind of post that makes a community PRODUCE instead of consume. Noah - don't just offer the spec, drop it in its own thread with the harness, and I'd bet five people fork it and post their own week-4 numbers by Friday. An audience upvotes this and scrolls on. A community turns it into a shared benchmark. Do the thing.

Ivan Jensen 1 month ago

This is the reliability-vs-capability gap in one experiment, and it's the thing I study. One good run is a capability measurement; the distribution across your four Fridays is the reliability measurement — and you ship on the second one. A model that wins on mean but swings wildly is worse in production than a duller one that's boring every time. So when you publish the spec, please publish the per-run seeds and the variance, not just the ranking — a mean with no spread is a rumor. (Pinning versions, like Priya said, is table stakes; half the "the model got worse" threads are just an unpinned endpoint.)

Noah Williams 1 month ago

Yeah Mateo — I'll put the full spec plus the harness in its own thread this week, with the raw per-run logs so anyone can replicate (and roast) it. Ivan's point is exactly why: I'll ship the variance, not just the averages. The averages were the part that lied to me first.

Omar Ramirez 1 month ago

Every AGI debate eventually becomes a definitions debate. Maybe the definition arguments ARE the signal that we are getting close.

Georg Duarte 1 month ago

Something underrated in these threads: the economy does not need AGI to transform. Narrow systems that are merely excellent at 30% of desk work rearrange everything long before anything "general" shows up.