Anthropic's extravagant tokenizer complicates AI pricing
There's no set definition of a token, but they're typically a set of three or four characters that are mapped to the integers LLMs actually process. Tokens have become the basic economic unit for billing use of AI models. Because the slicing of words into tokens and the tokens required per task vary across models, it has become rather difficult to predict the final bill for playing the AI slot machine.
- ▪There's no set definition of a token, but they're typically a set of three or four characters that are mapped to the integers LLMs actually process.
- ▪Tokens have become the basic economic unit for billing use of AI models.
- ▪Because the slicing of words into tokens and the tokens required per task vary across models, it has become rather difficult to predict the final bill for playing the AI slot machine.
Opening excerpt (first ~120 words) tap to expand
(function() { let windowUrl = window.location.href; windowUrl = windowUrl.substring(windowUrl.indexOf('?') + 1); let messageElement = document.querySelector('.shareableMessage'); if (windowUrl && windowUrl.includes('code') && windowUrl.includes('expires')) { messageElement.style.display = 'block'; } })(); AI and ML Anthropic's extravagant tokenizer complicates AI pricing Token consumption doesn't tell the whole tale but it shouldn't be ignored Thomas Claburn Thomas Claburn AI AND SOFTWARE REPORTER Senior reporter Published tue 14 Jul 2026 // 07:28 UTC Claude looks substantially more token-hungry than OpenAI's GPT-5.x, thanks to the new tokenizer that Anthropic shipped with recent releases.Large language models (LLMs) use tokenizers to handle the mapping of text into tokens.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at theregister.