Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
Kog AI has launched a tech preview of the Kog Inference Engine, achieving 3,000 tokens per second on AMD GPUs. This innovation aims to optimize AI inference speed on standard datacenter GPUs, addressing software bottlenecks that have limited performance. The focus on single-request decoding speed is expected to enhance the productivity of autonomous AI agents significantly.
- ▪Kog Inference Engine achieves 3,000 output tokens per second on AMD MI300X GPUs.
- ▪The technology aims to optimize AI inference speed on standard datacenter GPUs.
- ▪Single-request decoding speed is crucial for enhancing the productivity of autonomous AI agents.
Opening excerpt (first ~120 words) tap to expand
Inference Real-time LLM Inference on Standard GPUs (3,000 tokens/s per request) Today, Kog AI launches a tech preview of the Kog Inference Engine (KIE): 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). This preview runs a 2B model, with support for large third-party MoE models coming next at similar speeds. Kog Team 28 May 2026 — 16 min read Share (see below for full benchmark details) TL;DR: we show that AI inference on GPUs can be super-fast, reaching the speed regime of dedicated inference hardware cards when optimizing the whole software stack with architecture/engine/kernel co-design. Test the speed in our live coding playground: playground.kog.ai.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Kog Labs.
Discussion
0 commentsMore from Kog Labs
