KVSplit
AI
llm inference optimization
SHOW HN2025
272points
40comments
#206of 3,837 in 2025
Top 6%of 2025 by points
How HN took it
CuriousMostly questions and neutral comments
2 positive, 18 neutral or asking, 0 mixed, 12 critical, out of 32 comments about the project. Thread score -0.21.
Idea: 1 for, 3 againstExecution: 0 for, 3 againstPerformance: 0 for, 2 againstPrivacy: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting idea about asymmetric key/value precision
From 2 comments
Great work. This seems very interesting, but I need something slightly more high level to relate to it Will it just allow me to run let’s say a model with a 2048 token context…
nico on HNThis is a brilliant idea, and initiative. Does this also apply to GPUs? And I assume should be compatible with other quantization techniques, albeit they probably require their…
3abiton on HN
Objections
Same functionality already exists via --cache-type-k and --cache-type-v flags
From 4 comments
you can do this already with -ctk and -ctv, why would anyone need this? -ctk, --cache-type-k TYPE KV cache data type for K ``` allowed values: f32, f16, bf16, q8_0, q4_0, q4_1,…
segmondy on HNAm I missing something? As far as I can see this patch does nothing except add new options that replicate the functionality of the existing --cache-type-k and --cache-type-v…
therealsmith on HN> A note, your install script appears to still have a placeholder at the "apply patch" step. A suggestion, might be more user-friendly to fork llama.cpp and then include that as…
Aurornis on HNI finally had time to read the code. The patch is unnecessary because this functionality has been in llama.cpp since 2023 if I understand this PR correctly:…
Aurornis on HN
Install script applies a patch that is unnecessary and looks suspicious
From 3 comments
I finally had time to read the code. The patch is unnecessary because this functionality has been in llama.cpp since 2023 if I understand this PR correctly:…
Aurornis on HN> A note, your install script appears to still have a placeholder at the "apply patch" step. A suggestion, might be more user-friendly to fork llama.cpp and then include that as…
Aurornis on HNAfter looking more, this is definitely an AI driven attempt to game the system. It’s too bad the comments calling it out earlier were downvoted away. The first one was downvoted…
Aurornis on HN
0.86% perplexity loss is significant at short context
From 2 comments
+0.86% perplexity it's quite a bit at such a small context size though isn't it? How is it at more reasonable context sizes like 64-128k?
smcleod on HNYeah I get that, that's what we yse k/v cache quantisation for now which has a lower impact on PPL than this unless I'm missing something?
smcleod on HN
Maker's GitHub activity looks like LLM-generated star farming
From 2 comments
Finally someone making sense. The fact this project works by applying patches instead of forking the original project and committing changes should alone be reason for concern.…
imiric on HNAfter looking more, this is definitely an AI driven attempt to game the system. It’s too bad the comments calling it out earlier were downvoted away. The first one was downvoted…
Aurornis on HN
Requests
Explain how this differs from existing cache-type flags
From 2 comments
Is this any different from using --cache-type-k and --cache-type-v?
ondra on HNIs this any different from using --cache-type-k and --cache-type-v?
hiatus on HN
Fork llama.cpp instead of patch-apply install script
From 1 comment
Looks interesting! Is there any intuition for why this should be the case? Did you discover it via that intuition, or just random experimentation? A note, your install script…
matheist on HN
Benchmarks at longer contexts such as 64-128K
From 1 comment
+0.86% perplexity it's quite a bit at such a small context size though isn't it? How is it at more reasonable context sizes like 64-128k?
smcleod on HN
Support for MLX
From 1 comment
Is this patch possible to do on MLX? I'm getting better speeds on MLX. That, combined with your approach, would finally let Mac users have long conversations at usable speeds.
behnamoh on HN
Questions
Does this work on GGUF models without conversion?
From 1 comment
I'm curious: is it possible to apply differentiated KV quantization (like K8V4) to models after they're already converted to .gguf format, or does this require rebuilding the…
badmonster on HN
Does it speed up generation or only save memory?
From 2 comments
Amazing! Curious, what happens to performance? I assume you still pay the same performance price for longer context, even if you can now fit it in memory.
zmmmmm on HNWill that make the model run/feel faster? I can run models with 30-40b parameters on my computer, but they feel a lot slower than the 1-7b ones So would this make the 30-40b…
nico on HN
Does it require flash attention?
From 1 comment
I thought flash attention was required for quantised KV?
fennecbutt on HN
Alternatives named
llama.cpp --cache-type-k and --cache-type-v flags
From 2 comments
you can do this already with -ctk and -ctv, why would anyone need this? -ctk, --cache-type-k TYPE KV cache data type for K ``` allowed values: f32, f16, bf16, q8_0, q4_0, q4_1,…
segmondy on HNAm I missing something? As far as I can see this patch does nothing except add new options that replicate the functionality of the existing --cache-type-k and --cache-type-v…
therealsmith on HN
MLX
From 1 comment
Is this patch possible to do on MLX? I'm getting better speeds on MLX. That, combined with your approach, would finally let Mac users have long conversations at usable speeds.
behnamoh on HN