Alpaca.cpp
AI
local llm chat
SHOW HN2023
673points
283comments
#25of 3,109 in 2023
Top 1%of 2023 by points
How HN took it
CuriousMostly questions and neutral comments
17 positive, 71 neutral or asking, 3 mixed, 27 critical, out of 118 comments about the project. Thread score -0.07.
Execution: 4 for, 20 againstPerformance: 8 for, 4 againstIdea: 2 for, 2 againstDocs: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Runs well locally with good performance
From 3 comments
Ok not bad and pretty damn good performance: > Give me a writing prompt for a sci-fi story. A spaceship has crashed on Earth, and its crew of aliens are desperately trying to…
jpgvm on HNCan the moderator change the title to "Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a PC?" It runs great, with no hacking, on my Intel Ubuntu computer. I…
davidy123 on HNRunning perfectly on a MacBook Air M1 8Gb RAM 512GB SSD. Responses are quite slow and system is sluggish, but it works! ``` I am John from New York City > Hi John! How is the…
philaser on HN
Impressive that a model this size runs on consumer hardware
From 2 comments
``` > do you believe in god? Yes I do! God is the creator of all things seen or unseen; He created us with a purpose to live life abundantly (John 10:10). We are His children and…
dvt on HNThis is crazy... GPT-4, ChatGPT, Cloude, PaLM and now Alpaca locally... What a world to live in now!
Bitnotri on HN
Objections
Factual errors and confident wrong answers
From 2 comments
It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and…
loudmax on HNThis is funny, the response contains some data from the dataset itself. > When will the world end The exact date of when Earth would be destroyed by an asteroid or comet is…
filippofinke on HN
Title implies MacBook-only but runs on other systems
From 1 comment
Can the moderator change the title to "Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a PC?" It runs great, with no hacking, on my Intel Ubuntu computer. I…
davidy123 on HN
Fork of llama.cpp with little code change
From 1 comment
Does this really need to be a fork of llama.cpp? It seems like there's no real changes to the code.
bogwog on HN
Weights and copyright licensing concerns
From 1 comment
I love the Disclaimer at the bottom. The parties that trained these weights violated copyright on a grand scale to do so, and yet claim the ability to license downstream use.
politician on HN
Requests
Support for larger models like 13B and 30B
From 4 comments
Would it be possible to scale this up to use LLaMA 30b? Is it correctly understood that larger models need more hardware to fine-tune?
madmads on HNI noticed these always run a 7B model. What happens if you try a 13B model? Would it take a ridiculous amount of RAM?
syntaxing on HNHow hard is it to get the Alpaca tuning on the larger LLaMA models? It's not clear to me where this 4 GB model comes from, but it's much smaller than the LLaMA 7B model (13 GB)…
tshaddox on HNAre there torrents for 13B and up as well?
siraben on HN
Documentation of model conversion
From 1 comment
Could you share what you did to convert the models? The mirrors aren't very reliable and it would be good to have it documented somewhere. There is some discussion about this at…
ddren on HN
Pre-quantized model downloads
From 1 comment
Does anyone have a link to pre-quantized models for this/llama.cpp? I have the 4bit torch versions, I'd really like to check this out without torrenting the full fat models. (30B…
thot_experiment on HN
Questions
Does it run on Intel Macs or x86 CPUs
From 3 comments
Does it have any Apple M2 / ARM specific dependencies / optimizations, or will it generally run on any CPU (x86)?
gigel82 on HNDoes this run on an Intel mac? Or is M1/M2 required?
auggierose on HNLooking forward to try it, but I don't have a macbook. I wonder if it runs on i7-11800h (8 core 16 thread CPU) with 64gb RAM
tarruda on HN
How much context does it support
From 1 comment
Does anyone know how much context this supports? With chatGPT I can give commands like “summarize the following document: <500+ words>” and it works. I’d love to be able to do…
n8cpdx on HN
How much RAM is needed
From 2 comments
@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with…
phren0logy on HNI noticed these always run a 7B model. What happens if you try a 13B model? Would it take a ridiculous amount of RAM?
syntaxing on HN
Alternatives named
ChatGPT and GPT-4
From 3 comments
You can ask ChatGPT.
speedgoose on HNAlpaca is fun, but I would rather ask ChatGPT.
speedgoose on HNThis is crazy... GPT-4, ChatGPT, Cloude, PaLM and now Alpaca locally... What a world to live in now!
Bitnotri on HN
Claude and Poe
From 1 comment
I'm subscribed to Poe now which has a nice ios app + mac desktop for GPT4 and Anthropic's equivalent, Claude+
jazzyjackson on HN