Apple's On-Device Model is Terrible for Chat But Surprisingly Good at Structured Output and Tool Calling
Apple's on-device 3B parameter model performs poorly as a chatbot due to limited context length and generic responses, but excels at structured output and tool calling thanks to constrained decoding mechanisms in Swift. The model reliably generates valid JSON outputs and accurately decides when to invoke tools versus answering directly, despite its small size and local execution. Image generation through Image Playground works well for simple visuals but struggles with complex details or text rendering.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3906301) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Fernando Rodriguez Posted on Apr 30 • Originally published at frr.dev Apple's On-Device Model is Terrible for Chat But Surprisingly Good at Structured Output and Tool Calling #swift #swiftui #appleintelligence #macos I've spent weeks stress-testing Apple's on-device model — the ~3B parameter one that runs on the Neural Engine of any Apple Silicon Mac.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
