Running a 28.9M parameter LLM on an $8 microcontroller
Running a 28.9M parameter LLM on an $8 microcontroller Open to Work · 𝕏 slvDev · LinkedIn This is a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8. It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second. The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more.
- ▪Running a 28.9M parameter LLM on an $8 microcontroller Open to Work · 𝕏 slvDev · LinkedIn This is a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8.
- ▪It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second.
- ▪The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more.
Hacker News (AI / LLM) files mainly under ai. We currently carry 2,283 of its stories.
Opening excerpt (first ~120 words) tap to expand
Running a 28.9M parameter LLM on an $8 microcontroller Open to Work · 𝕏 slvDev · LinkedIn This is a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8. It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second. The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more. It fits because most of the model lives in flash instead of RAM, using an idea from Google's Gemma models called Per-Layer Embeddings.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Hacker News - Newest: ""AI" "LLM"".
Discussion
0 commentsMore from Hacker News - Newest: ""AI" "LLM""
