Deploying inference endpoints with PD disaggregation on AMD GPUs
dstack has introduced native support for Prefill-Decode (PD) disaggregation on AMD GPUs, enhancing its capabilities for deploying inference endpoints. This feature allows prefill and decode processes to run as separate pools, optimizing performance by addressing different bottlenecks. The deployment process involves configuring a service with specific requirements for high-bandwidth interconnects and resource allocation.
- ▪dstack is an open-source orchestrator that supports AI workloads across various environments.
- ▪PD disaggregation allows for independent scaling of prefill and decode processes, improving efficiency.
- ▪The deployment on AMD GPUs requires a high-bandwidth interconnect, such as RDMA over InfiniBand.
Opening excerpt (first ~120 words) tap to expand
Deploying inference endpoints with PD disaggregation on AMD GPUs¶ dstack is an open-source, AI-native orchestrator that works across clouds, Kubernetes clusters, on-prem fleets, hardware vendors, and frameworks. Alongside training, inference is one of the primary use cases dstack supports out of the box. dstack recently added native support for Prefill–Decode (PD) disaggregation. It works with Shepherd Model Gateway (SMG) — a high-performance inference gateway evolved from the SGLang Router — on both NVIDIA and AMD, and with NVIDIA Dynamo on NVIDIA. This post walks through deploying it on AMD GPUs with SMG. Why PD disaggregation¶ PD disaggregation is useful when a single LLM deployment has two different bottlenecks: Prefill processes the prompt.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Hacker News (Newest).
Discussion
0 commentsMore from Hacker News (Newest)
