Outlines
AI
constrained llm text generation
SHOW HN2023
854points
303comments
#10of 3,109 in 2023
Top 1%of 2023 by points
How HN took it
CuriousMostly questions and neutral comments
20 positive, 63 neutral or asking, 2 mixed, 30 critical, out of 115 comments about the project. Thread score -0.11.
Idea: 12 for, 21 againstPerformance: 4 for, 4 againstExecution: 0 for, 3 againstDocs: 1 for, 2 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Masking approach is efficient, with index built once and fast per-step lookups
From 2 comments
Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index…
remilouf on HNTwo differences: (1) This feature only requires regex-guided generation. We have a PR for BNF sampling that is about to be merged. (2) ggml loops over the entire vocabulary (~50k…
remilouf on HN
Objections
Masking may distort the model's probability distribution and hurt output quality
From 2 comments
A major part of the power of an LLM is the calibrated probability distribution in its responses, and this technique probably throws that ability away. Why is it good enough? As a…
hansvm on HNThis is exciting, we built a similar tool[1] recently specifically targeted at constraining llama output to match a TypeScript interface. I firmly believe that output format…
aduffy on HN
Valid JSON does not mean correct content
From 2 comments
OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form…
Animats on HN"Generating valid JSON" is not impressive. Here's some valid JSON: [] The tricky part is generating useful JSON.
tantalor on HN
Prior similar libraries already exist
From 2 comments
I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or…
BoorishBears on HNThe “trick” seems to blatantly rip off FlashText without citing it? https://arxiv.org/pdf/1711.00046.pdf I’m a fan of the approach. I normally wouldn’t care if this was just…
lefttoreader on HN
Requests
Integration with llama.cpp
From 1 comment
Can I use this locally with models that run on my CPU? Like llama.cpp
haolez on HN
Tutorial for running locally with llama2
From 2 comments
Would love to have a tutorial on how to install and run this locally with a nice model, for those of us who are behind the 8-ball with torch, transformers, diffusers, llama2 etc.
dvasdekis on HNAs a more general comment, the repo README provides examples that all use gpt2. It would be nice to see at least one example that invokes llama2, since I feel like that would…
coder543 on HN
Questions
Does it work with GPT-4 or OpenAI's API
From 2 comments
Does this work with GPT-4?
nikcheerla on HNI really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format)…
simonw on HN
How much latency does it add
From 2 comments
Very cool! How much latency does it add?
huevosabio on HNI tried slight modifications from the example pydantic model and it's incredibly slow. Maybe I'm doing something wrong but I've a hefty box and a 3090, an example using gpt-2…
IanCal on HN
Alternatives named
jsonformer
From 2 comments
I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or…
BoorishBears on HNHow does this compare in terms of latency, cost, and effectiveness to jsonformer? https://github.com/1rgs/jsonformer
leetharris on HN
guidance
From 2 comments
I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or…
BoorishBears on HNit still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly, in fact, it does not even…
2bitencryption on HN
llama.cpp grammar sampling
From 2 comments
Relevant; LLama.cpp implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773
sneedchucker on HNHow does this relate to ggmls bnf sampling?
spott on HN
TypeChat
From 1 comment
Looks interesting! How would you say it compares to Microsoft's TypeChat (beyond the obvious Python/TypeScript difference)?…
Deukhoofd on HN