felafax
AI
llm fine-tuning infrastructure
SHOW HN2024
189points
52comments
#422of 3,515 in 2024
Top 13%of 2024 by points
How HN took it
CuriousMostly questions and neutral comments
2 positive, 16 neutral or asking, 0 mixed, 7 critical, out of 25 comments about the project. Thread score -0.14.
Pricing: 1 for, 3 againstPerformance: 0 for, 3 againstDesign: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Cost-effective alternative to NVIDIA GPUs
From 2 comments
Runpod charges $3.49/hr for an H100 SXM, which is fairly cheap as far as on-demand H100s go. A v5p TPU is $4.20/hr, but has 95GB RAM instead of 80GB on the H100 — so you'll need…
reissbaker on HNIt took roughly 2-3 weeks to translate Torch to JAX, but I had past experience writing JAX from my time at Google. We spent nearly 4 weeks getting PyTorch XLA working on TPU.…
felarof on HN
Objections
Cost comparison against NVIDIA is not apples-to-apples and was not run on NVIDIA
From 2 comments
Then how did you compute the 30-70% cost efficiency numbers compared to Nvidia if you haven’t run this Llama finetuning task on Nvidia GPUs?
p1esk on HNIt would be a lot more convincing if you actually ran it yourself and did a proper apples to apples comparison, especially considering that’s the whole idea behind your project.
p1esk on HN
Google Cloud GPU prices are inflated compared with other providers
From 3 comments
It's also comparing prices on google cloud, which has its own markup, a lot more expensive than say runpod. Runpod is $1.64/hr for the A100 on secure cloud while the A100 on…
KaoruAoiShiho on HN> For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs When you say this, you should specify which Nvidia GPU you mean (I assume h100…
Palmik on HNWouldn't really say it's the cheapest option...there are other providers like Lambda Labs or Ori.co where you can find them way cheaper
danvdb on HN
No LoRA support in the JAX path
From 1 comment
Requests
Add LoRA training support in JAX
From 2 comments
Very cool! Unlocking TPU training is a big win. FWIW, if this helps prioritize: personally I'd find LoRA training for Llama 3.1 most useful (which it sounds like currently isn't…
reissbaker on HNThanks for the detailed feedback! Yes, supporting LoRA fine-tuning is one of the things we are already working on. btw, we have LoRA supported with Llama3 PyTorch-XLA model.…
felarof on HN
Questions
How the 30-70% cost savings were computed
From 2 comments
Did you try running this task (finetuning Llama) on Nvidia GPUs? If yes, can you provide details (which cloud instance and time)? I’m curious about your reported 30-70% speedup.
p1esk on HNThen how did you compute the 30-70% cost efficiency numbers compared to Nvidia if you haven’t run this Llama finetuning task on Nvidia GPUs?
p1esk on HN
Speed comparison with Unsloth
From 2 comments
Anyone want to comment on this versus the fine tune speedups from llama3.1 with unsloth?
xrd on HNWhat was the estimate for how much time you guys took to translate the torch to Jax vs how much you spent on XLA?
htrp on HN
Alternatives named
Unsloth
Runpod
From 1 comment
Runpod charges $3.49/hr for an H100 SXM, which is fairly cheap as far as on-demand H100s go. A v5p TPU is $4.20/hr, but has 95GB RAM instead of 80GB on the H100 — so you'll need…
reissbaker on HN
Lambda Labs
From 2 comments
Wouldn't really say it's the cheapest option...there are other providers like Lambda Labs or Ori.co where you can find them way cheaper
danvdb on HNRunpod charges $3.49/hr for an H100 SXM, which is fairly cheap as far as on-demand H100s go. A v5p TPU is $4.20/hr, but has 95GB RAM instead of 80GB on the H100 — so you'll need…
reissbaker on HN