The Benchmark-pocalypse

There’s been a lot of talk about the vuln-pocalypse, to which Dan Luu doesn’t have much to add because Dan isn’t not a security person. But there hasn’t been much discussion on the closely related (and to be fair, less serious, issue), the benchmark-pocalypse.

While it’s become easier than ever to make serious performance gains, it’s also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of “we rewrote X in Rust” project or a new startup that’s looking to either fundraise or sell something, but it happens on other kinds of projects as well.

Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It’s always been easy to fake up an unrepresentative microbenchmark and that’s never going to change. What’s changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler “optimizations” that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs not only make this trivial, they do it by default, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did.

Rather than point to someone’s bad claim, I’ll point to FRE, this regex engine I had an agent build, which I could claim is the world’s fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster on rebar. But agents are wont to reward hack and overfit unless you put serious guardrails in place to avoid that, which I didn’t do in this case as an experiment.

See more on danluu.com. Via Hacker News.

from Adafruit Industries – Makers, hackers, artists, designers and engineers! https://ift.tt/Edaro8i
via IFTTT