A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
AI
on-chip llm inference
SHOW HN2026
A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
mikeayles.com#fpga inference#on-chip llm#kv260#sram inference#tokens per second
How HN took it
5 positive, 5 neutral or asking, 1 mixed, 1 critical, out of 12 comments about the project. Thread score +0.27.
Idea: 3 for, 1 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Conceptually interesting and well documented
From 2 comments
Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds/token and you have this wave of optimism like "It's the effort that…
tomex on HNAmazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved,…
gbxk on HN
Live demo is great
From 1 comment
This is so cool, and having a live demo is just chefs kiss
anitil on HN
Objections
Model output too small to be useful; gibberish in chat
From 2 comments
Very nice indeed. Model weights in RAM blocks distributed all over a big FPGA: should be super helpful at minimizing RAM bandwidth bottlenecks. To say nothing of latency. But…
RetroTechie on HNconceptually it's a cool idea. practically the results seem about as coherent as ``` import random; print(random.choice(list(my_dict))) ``` ..but way slower is there a practical…
serf on HN
FPGAs are not power efficient compared to GPUs or ASICs
From 2 comments
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
cgyvbunji on HNNothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of…
adrian_b on HN
FPGA design is hard to learn and niche
From 1 comment
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and…
stevefan1999 on HN
Requests
Try a bigger FPGA or a larger model
From 2 comments
Very nice indeed. Model weights in RAM blocks distributed all over a big FPGA: should be super helpful at minimizing RAM bandwidth bottlenecks. To say nothing of latency. But…
RetroTechie on HNAmazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved,…
gbxk on HN
Partial reconfiguration to avoid full stalls during model swaps
From 1 comment
does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request…
useiris on HN
Questions
Is there a practical use for a model this small?
From 1 comment
conceptually it's a cool idea. practically the results seem about as coherent as ``` import random; print(random.choice(list(my_dict))) ``` ..but way slower is there a practical…
serf on HN
What is per-user latency under many concurrent connections?
From 1 comment
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
haeseong on HN
Alternatives named
Taalas chip demo (Llama 8B in custom silicon)
From 1 comment
I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that…
mikeayles on HN
Cactus compute runtime
From 1 comment
Amazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved,…
gbxk on HN