ntransformer
AI
local llm inference
SHOW HN2026
How HN took it
7 positive, 28 neutral or asking, 0 mixed, 10 critical, out of 45 comments about the project. Thread score -0.01.
Performance: 2 for, 8 againstIdea: 4 for, 1 againstPricing: 0 for, 1 againstExecution: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Impressive systems engineering with tiered caching
From 1 comment
This is a really impressive piece of systems engineering. The 3-tier adaptive caching (VRAM resident > pinned RAM > NVMe/mmap) is essentially reimplementing what the Linux…
turingsroot on HN
Clever bypass of CPU for NVMe-to-GPU transfer
From 1 comment
Bypassing CPU for NVMe-to-GPU transfer is clever. The bottleneck for running large models locally has always been the memory hierarchy — this essentially treats NVMe as extended…
MarcLore on HN
Objections
Around 0.2 to 0.5 tokens per second is too slow for interactive use
From 2 comments
0.2 tok/s is fine for experimentation, but it is not interactive in any meaningful sense. For many use cases, a well-quantized 8B or 13B that stays resident will simply deliver a…
randomtoast on HNCool hack but 0.5 tok/s on 70B when a 7B does 30+ on the same card. NVIDIA's own research says 40-70% of agentic tasks could run on sub-10B models and the quality gap has closed…
7777777phil on HN
Smaller quantized models stay resident and perform better
From 2 comments
0.2 tok/s is fine for experimentation, but it is not interactive in any meaningful sense. For many use cases, a well-quantized 8B or 13B that stays resident will simply deliver a…
randomtoast on HNCool hack but 0.5 tok/s on 70B when a 7B does 30+ on the same card. NVIDIA's own research says 40-70% of agentic tasks could run on sub-10B models and the quality gap has closed…
7777777phil on HN
Cost per token is poor compared with hosted APIs
From 1 comment
Cost wise it does not seem very effective. .5 token / sec (the optimized one) is 3600 tokens an hour, which costs about 200-300 watts for an active 3090+system. Running 3600…
esquire_900 on HN
Requests
Benchmarks on Gen5 NVMe or faster PCIe links
From 2 comments
This is a really impressive piece of systems engineering. The 3-tier adaptive caching (VRAM resident > pinned RAM > NVMe/mmap) is essentially reimplementing what the Linux…
turingsroot on HNReally interesting experiment i should have done this before Do you have numbers on effective throughput vs PCIe theoretical bandwidth? I’m curious whether this is primarily…
davideom0414 on HN
Comparison with Apple unified memory M-series
From 1 comment
Bypassing CPU for NVMe-to-GPU transfer is clever. The bottleneck for running large models locally has always been the memory hierarchy — this essentially treats NVMe as extended…
MarcLore on HN
More details on the DKMS patching process for consumer GPUs
From 1 comment
Cool project. Can you provide more details about your DKMS patching process for consumer GPUs? This would be fun to try out, but I’d need some more details on that patch process…
Aurornis on HN
Questions
Is throughput latency-bound or bandwidth-bound
From 1 comment
Really interesting experiment i should have done this before Do you have numbers on effective throughput vs PCIe theoretical bandwidth? I’m curious whether this is primarily…
davideom0414 on HN
Can AMD GPUs do direct IO like this
From 1 comment
Interesting. Can AMD GPUs do direct io like this?
stuaxo on HN
Alternatives named
llama.cpp with CPU and GPU layer offload
From 2 comments
That's slower than just running it off CPU+GPU. I can easily hit 1.5 tokens/s on a 7950X+3090 and a 20480-token context.
fluoridation on HNSo llama.cpp currently statically puts overflow MoE experts in RAM and inferences them on CPU, so you get a mix of GPU + CPU inferencing. You are rooflined by RAM->CPU bandwidth…
rao-v on HN
Running smaller 7B to 8B models fully in VRAM
From 2 comments
Cool hack but 0.5 tok/s on 70B when a 7B does 30+ on the same card. NVIDIA's own research says 40-70% of agentic tasks could run on sub-10B models and the quality gap has closed…
7777777phil on HN0.2 tok/s is fine for experimentation, but it is not interactive in any meaningful sense. For many use cases, a well-quantized 8B or 13B that stays resident will simply deliver a…
randomtoast on HN