I Blamed the Model for Months. The Bug Was My Sampler.
The author discusses their experience running a local language model on an M1 Max machine. Initially, they blamed the model's architecture for poor output quality, but later discovered that the issue stemmed from a flawed sampler configuration in their code. After making adjustments, the model's performance improved significantly, demonstrating the importance of proper configuration in machine learning applications.
- ▪The author ran a 35B Mixture-of-Experts model that consumed around 40GB of memory.
- ▪They initially assumed the model's architecture was the problem due to poor output quality.
- ▪After diagnosing the issue, they found that a custom repetition penalty processor was causing incoherence in the generated text.
- ▪By changing the model and adjusting the sampler configuration, the output quality improved dramatically.
- ▪The system's free memory increased significantly after switching to a smaller model, enhancing overall performance.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3885340) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } SleepyQuant Posted on May 29 • Originally published at sleepyquant.rest I Blamed the Model for Months. The Bug Was My Sampler. #applesilicon #mlx #localai #m1max I Blamed the Model for Months. The Bug Was My Sampler. 40GB In, Word Salad Out Running local LLMs on M1 Max hardware is one of those setups that looks great on paper — unified memory, no PCIe bottleneck, offline and private.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
