Cactus Hybrid
AI
on-device language model confidence
SHOW HN2026
How HN took it
6 positive, 10 neutral or asking, 0 mixed, 2 critical, out of 18 comments about the project. Thread score +0.15.
Idea: 2 for, 1 againstExecution: 2 for, 0 againstDocs: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting idea and useful on-device approach
From 2 comments
Cool idea, some feedback: 1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This…
dmrivers on HNLove what you're doing at cactus, keep it up!!
asar on HN
Objections
Calling it knowing when wrong overstates what a confidence score can show
From 1 comment
> "post-trained to know when it's wrong" Is it also post-trained to know when it's wrong about whether it's wrong? > "Every response comes with a confidence score between 0 and…
BugsJustFindMe on HN
Probe may be unreliable; rerunning with a new seed seems simpler
From 1 comment
I don't really get why you need handoff if your score is accurate. If it is, and it is low for a given response, just let the harness re-run the prompt with a different seed…
maxgashkov on HN
Rerunning the model for consistency drains battery on edge devices
From 1 comment
1. Thanks for this! 2. We combine both on Cactus; rolling entropy method helps with per-token signal for early exit, while the probe runs on complete output. 3. Users hate this…
HenryNdubuaku on HN
Requests
Benchmark on coding tasks with a larger local model fallback
From 1 comment
Have you benched this for coding tasks, with a fallback to a larger local model, for example Qwen-3.6-27B? Or using it for sub-tasks, where a framework with a larger primary…
zdw on HN
Compare against conformal prediction and consistency-based methods (AUROC)
From 1 comment
Cool idea, some feedback: 1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This…
dmrivers on HN
Questions
Does model quality degrade with the added weights?
From 2 comments
Does the model quality become degraded in other ways?
robrenaud on HNIt's impossible to add weights which degrade the quality?
zamadatix on HN
How the probe is trained on hidden states to predict p(wrong)
From 1 comment
> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various…
cacio-e-pepe on HN
Alternatives named
Goodfire's RLFR work
From 1 comment
Is this in any way similar to Goodfire's work? https://www.goodfire.ai/research/rlfr#
astrobiased on HN