LlamaGPT
AI
self-hosted chatbot
SHOW HN2023
How HN took it
7 positive, 22 neutral or asking, 0 mixed, 5 critical, out of 34 comments about the project. Thread score +0.08.
Idea: 4 for, 3 againstExecution: 2 for, 0 againstPerformance: 0 for, 2 againstPrivacy: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Simple one-command setup makes local LLMs accessible
From 2 comments
The main difference is setting everything up yourself manually, downloading the modal, optimizing the parameters for best performance, running an API server and a UI front-end -…
mayankchhabra on HNAgreed. Gpt4all[1] offers a similar 'simple setup' but with application exe downloads, but is arguably more like open core because the gpt4all makers (nomic?) want to sell you…
Multicomp on HN
Private and offline operation appeals to users
From 1 comment
I've been looking for something like this for a while. Nice!
QuinnyPig on HN
Objections
GPU support is not available yet; CPU inference is slow
From 2 comments
Nice project! I could not find the information in the README.md, can I run this with a GPU? If so what do I need to change? Seems like it's hardcoded to 0 in the run script:…
belval on HNGood luck! The token/sec will be under your expectations or it will overheat. You really shouldn't play games with your data-storage. You could try it with an old laptop to see…
reneberlin on HN
Unclear what it adds over llama.cpp or existing installers like oobabooga
From 2 comments
I didn't see any info on how this is different than installing/running llamacpp or koboldcpp. New offerings are awesome of course, but what is it adding?
SubiculumCode on HNMaybe I've been at this for too long and can't see the pitfalls of a normal user, but how is that easier than using an oobabooga one-click installer (an option that's been around…
DrPhish on HN
Name uses GPT, a possible trademark issue
From 1 comment
So many projects still using GPT in their name. Is the thinking here that OpenAI is not going to defend that trademark? Or just kicking the can down the road on rebranding until…
caesil on HN
Requests
GPU support (CUDA and Metal)
From 2 comments
Ah yes, running on GPU isn't supported at the moment. But CUDA (for Nvidia GPUs) and Metal support is on the roadmap!
mayankchhabra on HNI put up a draft PR to demo how to run it on a GPU: https://github.com/getumbrel/llama-gpt/pull/11 It breaks other things like model downloading, but once I got it to a working…
dicriseg on HN
Ability to add private data such as emails or documents
From 1 comment
Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.
ccozan on HN
Guide for non-ML users on adding private data
From 1 comment
Thanks for all replies. It would probably be worth creating a HOWTO or something like that aimed at non ML experts or complete AI illiterates like myself to help putting together…
squarefoot on HN
Questions
How to import models from Hugging Face
From 1 comment
thanks. yeah, that IS useful. Anyone see if it contains utilities to import models from huggingface/github?
SubiculumCode on HN
Whether the model is censored
From 2 comments
is it a free model or is the politically-correct-only response constraints in place?
chasd00 on HNLlama is definitely "censored" though I've not found this to be an issue in practice. Guess it depends on what you want to do with it
Havoc on HN
Whether it runs on a cloud server or NAS
From 2 comments
Oh I thought this was a quick guide to host it on any server (AWS / other clouds) of our choosing.
albert_e on HNInteresting. I might try to get this to work on my NAS.
ryanSrich on HN
Alternatives named
Ollama
From 1 comment
https://github.com/jmorganca/ollama was extremely simple to get running on my M1 and has a couple uncensored models you can just download and use.
lkbm on HN
Serge
From 1 comment
Very cool, this looks like a combination of chatbot-ui and llama-cpp-python? A similar project I've been using is https://github.com/serge-chat/serge. Nous-Hermes-Llama2-13b is…
Atlas-Marbles on HN
GPT4All
From 1 comment
Agreed. Gpt4all[1] offers a similar 'simple setup' but with application exe downloads, but is arguably more like open core because the gpt4all makers (nomic?) want to sell you…
Multicomp on HN
oobabooga text-generation-webui
From 1 comment
Maybe I've been at this for too long and can't see the pitfalls of a normal user, but how is that easier than using an oobabooga one-click installer (an option that's been around…
DrPhish on HN