The same LLM is 8x slower to first token depending on who serves it
Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all. The setup Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting.
- ▪Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn
- ▪The setup Two backends for the identical model.
- ▪A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting.
Opening excerpt (first ~120 words) tap to expand
We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all. The setup Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a stream: true request and return an SSE stream. On paper, interchangeable.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Dynoyard.
Discussion
0 commentsMore from Dynoyard
