Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding
Cassandra is a new algorithm-hardware co-designed framework aimed at improving the efficiency of reasoning large language models (LLMs) through self-speculative decoding. It addresses the challenges of decode-stage overhead and accuracy degradation in existing methods, achieving significant speedups without additional training. Experimental results demonstrate that Cassandra outperforms state-of-the-art methods, generating more tokens under the same memory constraints.
- ▪Cassandra is designed to enhance the performance of reasoning LLMs by utilizing a self-speculative decoding framework.
- ▪The framework achieves up to 2.41x speedup over the BF16 baseline and generates 1.81x more tokens compared to the Eagle-3 method.
- ▪Cassandra employs fine-grained data selection and optimized pruning to improve efficiency without requiring additional training.
Opening excerpt (first ~120 words) tap to expand
Computer Science > Hardware Architecture arXiv:2605.26558 (cs) [Submitted on 26 May 2026] Title:Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding Authors:Soongyu Choi, Yuntae Kim, Muyoung Son, Joo-Young Kim View a PDF of the paper titled Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding, by Soongyu Choi and 3 other authors View PDF HTML (experimental) Abstract:Speculative decoding has emerged as a promising lossless approach for accelerating Large Language Models (LLMs). As reasoning LLMs increasingly suffer from decode-stage overhead and approximation-based methods degrade accuracy, lossless speculative decoding has become essential for efficient inference.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at arXiv.org.
Discussion
0 commentsMore from arXiv.org
