Running a 28.9M parameter LLM on an $8 microcontroller

Slava S. on GitHub has a 28.9 million parameter language model that generates text on an ESP32-S3, a microcontroller that costs about $8.

It runs on the chip itself, with nothing sent to a server, and it writes each word to a small screen wired to the chip at roughly 9 tokens per second.

The last language model people ran on a chip like this had 260 thousand parameters, so this one holds about a hundred times more. It fits because most of the model lives in flash instead of RAM, using an idea from Google’s Gemma models called Per-Layer Embeddings.

The way around it is to stop putting the model in fast memory at all. Most of a language model’s parameters sit in an embedding table, which the model reads from rather than computes on. So you can leave that 25 million row table in slow flash and pull only the few rows each token needs, about 450 bytes, while the small part that does the actual work stays in fast memory. The large model then costs almost nothing to run, because you never load most of it. It just sits in flash and gets sampled a little at a time.

That idea is Google’s Per-Layer Embeddings, from Gemma 3n and Gemma 4. Here it runs on the memory layout of a microcontroller instead of a phone or a GPU. As far as I can tell, nobody had tried it on a chip this small.

  SRAM  (fast, tiny)   the "thinking" core, used on every token
  PSRAM (medium)       the output head and working memory
  FLASH (huge, slow)   the 25M-param table, 6 rows per token (~450 B)

The model was trained on TinyStories, so it writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small part of the model that does the reasoning.

See more on this MIT licensed project on GitHub.

from Adafruit Industries – Makers, hackers, artists, designers and engineers! https://ift.tt/83cvtdT
via IFTTT