{"id":160,"date":"2026-10-16T23:31:29","date_gmt":"2026-10-16T23:31:29","guid":{"rendered":"https:\/\/llama.gs\/blog\/?p=160"},"modified":"2026-10-16T23:41:33","modified_gmt":"2026-10-16T23:41:33","slug":"the-rise-and-fall-of-benchmarking","status":"publish","type":"post","link":"https:\/\/llama.gs\/blog\/index.php\/2026\/10\/16\/the-rise-and-fall-of-benchmarking\/","title":{"rendered":"The Rise and Fall of Benchmarking"},"content":{"rendered":"\n

It is Friday. The French used to say \u201cLe roi est mort, vive le roi!\u201d. For those of us raised with less melodious languages, this means \u201cThe king is dead, long live the king!\u201d And since we don\u2019t use guillotines to kill humans anymore, our brightest and most capable chant in media something alongside \u201cMathematics is dead. Long live mathematics.\u201d Keep in mind that computer science (and AI by extension) is just B-rated mathematics for people who didn\u2019t study for their exams because they spent too much time playing video games or fraternizing with their Greek-letter friends at business school.<\/p>\n\n\n\n

Now let\u2019s talk about open problems and OpenAI allegedly closing millennial mathematical problems, and scientists and other folks crying. Oh well, mathematicians cry because of all these suppressed emotions, but that is something for the shrinks to weigh-in and I am not a shrink (admittedly most of my knowledge about psychology comes from watching The Sopranos, reading Phillip Roth, and reading the Wikipedia article on Sigmund Freud).<\/p>\n\n\n\n

Back to OpenAI solving millennial problems. We like lists, don\u2019t we? It is very satisfying to check every item in a list. It sure beats bubble-wrap. Of course the proverbial million-dollar questions are: what is this all good for, can we (anybody) actually make sense of the results, and are the results any correct? Well, before solving millennial problems, opening the gates to million-dollar stipends, lecture halls full of applauding people and economy-plus-paid conference travel, there used to be all kinds of competitions and challenges. Remember the machine-learning Netflix challenge to predict movie watching (and the million-dollar prize)?<\/p>\n\n\n\n

Also, remember the Euler problem website where I spent 3-4 moderately happy do-nothing days during my Ph.D. years? One had to solve, mostly, number-theoretical problems of increasing difficulty, and one got some badges and stars and their name in a leaderboard, so one could boast to their geek friends. At some point, I solved all hundred problems, but soon after additional problems emerged, and I was bored, and I moved on, and the world moved on as well.<\/p>\n\n\n\n

Talking about problems and competitions, I spent a lot of time on a project called \u201cThe International Diagnostic Competition\u201d. This was collaboration between NASA and Xerox PARC, and I spent one year setting-up the competition, and PARC synthesized diagnostic challenges based on small chip designs, and NASA provided data about the electrical simulator they had. NASA\u2019s simulator was an expensive room somewhere around Moffet Field, full of fans and light bulbs and buttons (rumor went there was a solar panel on the roof).<\/p>\n\n\n\n

The details escape me, but I believe there were three submissions: one from me, one from my manager Johan, and one from Sweden (where the DX workshop was this year). I think my algorithm won, but that is probably because I generated the problem formulation and the problems and the metrics for ranking the solutions. I should have let Johan win because he was my PARC manager, and it would be better for my career and everything.<\/p>\n\n\n\n

The general conversation from those times though resonates in my head: your algorithm is faster but my uses less memory and yours has more false positives but mine has less false negatives and even second graders know false negatives are more important than false positive but you give permission for travel expenses and you are someone who influences the lab manager who badly wants to be a CEO so he can buy the cheapest Porsch so you win!<\/p>\n\n\n\n

Then there is the satisfiability competition. Boolean satisfiability is a well-defined computer science problem, and people agreed the CPU time matters most and that there is a single (process) thread and also a parallel SAT thread. But then somebody was clever enough to make use of cache memory in a better way or something, and is this true algorithmics (like B-rated mathematics) or is this a con job?<\/p>\n\n\n\n

Every so often I glimpse into the state of LLMs, and then I see a million benchmarks about translation and mathematical problem solving and everything else. Oh well, isn\u2019t the list with all open mathematical problems also a kind of benchmark? Don\u2019t worry! There will be a benchmark and a prize for everyone. If you won nothing, just invent a benchmark and a solution. And, well, maybe you can make your desired millions or billions. With this inflation, soon we will use wheelbarrows to carry money around. The photo below is from postwar Germany when inflation was so high that a loaf of bread cost a billion marks. The very inflation that the unsustained ambitions of Silicon Valley will fuel.<\/p>\n\n\n\n

\"\"
Inflation in the Weimar Republic: an artist using paper money as a cheap wallpaper substitute. Photo taken from Wikimedia.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

It is Friday. The French used to say \u201cLe roi est mort, vive le roi!\u201d. For those of us raised with less melodious languages, this means \u201cThe king is dead, long live the king!\u201d And since we don\u2019t use guillotines to kill humans anymore, our brightest and most capable chant in media something alongside \u201cMathematics […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-160","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/posts\/160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=160"}],"version-history":[{"count":2,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/posts\/160\/revisions"}],"predecessor-version":[{"id":163,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/posts\/160\/revisions\/163"}],"wp:attachment":[{"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/llama.gs\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}