From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
The paper introduces ChemCoTBench-V2, a benchmark designed for evaluating chemical reasoning in large language models. It emphasizes the importance of process-level evaluation rather than just final answers, highlighting the potential for models to provide correct outputs while lacking sound reasoning. The benchmark aims to facilitate fine-grained comparisons between models and identify specific failures in their reasoning processes.
- ▪ChemCoTBench-V2 is a rule-verifiable diagnostic benchmark for evaluating chemical reasoning in large language models.
- ▪The benchmark includes 5,620 evaluation samples across 18 tasks, focusing on molecular understanding and reaction prediction.
- ▪Experiments show a gap between final-answer correctness and structured reasoning, indicating models may fail in chemical-step checks.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Artificial Intelligence arXiv:2606.03660 (cs) [Submitted on 2 Jun 2026] Title:From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models Authors:Hongyu Guo, Hao Li, He Cao, Gongbo Zhang, Li Yuan View a PDF of the paper titled From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models, by Hongyu Guo and 4 other authors View PDF HTML (experimental) Abstract:Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv cs.AI.
Discussion
0 commentsMore from arXiv cs.AI
