Fitting WhisperX large-v3 + a 24B LLM on one 3090: a reproducible context-capping recipe
The article discusses a method for fitting WhisperX large-v3 and a 24B LLM on a single RTX 3090 GPU. It details the technical steps taken to avoid out-of-memory errors while maintaining performance. The author provides a reproducible recipe, including measurements and configurations for effective resource management.
- ▪The setup includes a 24GB RTX 3090, WhisperX large-v3 for speech-to-text, and a Devstral Small 24B LLM for email triage.
- ▪By capping the context window to 8192 tokens, the total memory usage was reduced to 21.9GB, preventing out-of-memory errors.
- ▪The author created a dashboard to monitor VRAM usage per service, which helped identify and resolve contention issues.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 1410108) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Arsen Apostolov Posted on Jun 3 Fitting WhisperX large-v3 + a 24B LLM on one 3090: a reproducible context-capping recipe #homelab #ollama #localllm #devops This is the technical, reproducible version of a fix I shipped on my own homelab. If you want the narrative version, that's on Medium. This one is the recipe: the measurements, the math, the Modelfile, and the exact prompt I gave Claude Code to generate it. Copy-paste friendly.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
