Life After Benchmark Saturation: A Case Study of CORE-Bench
arXiv:2606.26158v1 Announce Type: new Abstract: When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version. We show that this approach privileges accuracy and misses the opportunity to study six other key dimensions of agent performance: construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration. We use C
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2606.26158 (cs) [Submitted on 23 Jun 2026] Title:Life After Benchmark Saturation: A Case Study of CORE-Bench Authors:Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona, Stephan Rabanser, Tilman Bayer, Abhishek Shetty, Yue Ling, Derrick Chan-Sew, Rumi Nakagawa, Saiteja Utpala, Zachary S. Siegel, Arvind Narayanan View a PDF of the paper titled Life After Benchmark Saturation: A Case Study of CORE-Bench, by Nitya Nadgir and 13 other authors View PDF HTML (experimental) Abstract:When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv.org.
Discussion
0 commentsMore from arXiv.org
