Show HN: LLM Proxy - Python with SSE stream aggregation and timeout prevention
This lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a NVIDIA-hosted model without any client-side changes — you simply point the client at llmproxy instead of at a real local runtime. It covers chat, completions, and embeddings, supports streaming, multi-model discovery, optional inbound authentication, automatic retries on transient upstream errors, and a live /stats metrics & process dashboard. Demo The proxy starts, exposes the models, and answers both an OpenAI-compatible /v1/chat/completions call and a native Ollama streaming /api/chat call — every request forwarded to NVIDIA.
- ▪This lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a NVIDIA-hosted model without any client-side changes — you simply point the client at llmproxy instead of at a real local runtime.
- ▪It covers chat, completions, and embeddings, supports streaming, multi-model discovery, optional inbound authentication, automatic retries on transient upstream errors, and a live /stats metrics & process dashboard.
- ▪Demo The proxy starts, exposes the models, and answers both an OpenAI-compatible /v1/chat/completions call and a native Ollama streaming /api/chat call — every request forwarded to NVIDIA.
Hacker News (AI / LLM) files mainly under ai. We currently carry 2,122 of its stories.
Opening excerpt (first ~120 words) tap to expand
llmproxy A lightweight, high-performance LLM proxy for caching, automatic failover, cost tracking, and seamless integration between local and cloud AI providers. llmproxy is a lightweight Flask server that emulates the HTTP APIs of several popular local LLM runtimes (Ollama, the OpenAI /v1 API, and llama.cpp's llama-server) and transparently forwards every request to NVIDIA's OpenAI-compatible API (https://integrate.api.nvidia.com/v1). This lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a NVIDIA-hosted model without any client-side changes — you simply point the client at llmproxy instead of at a real local runtime.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.