PromptTools
Dev tools
llm evaluation testing
SHOW HN2023
211points
24comments
#264of 3,109 in 2023
Top 9%of 2023 by points
How HN took it
LovedMost comments that take a side are positive
5 positive, 6 neutral or asking, 2 mixed, 0 critical, out of 13 comments about the project. Thread score +0.34.
Idea: 2 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful tooling for the LLM evaluation problem
From 3 comments
Super cool, the need for tooling like this is something one realizes pretty quickly when starting to build apps that leverage LLMs.
catlover76 on HNSomething like this is going to be needed to evaluate models effectively. Evaluation should be integrated into automated pipelines/workflows that can scale across models and…
neelm on HNThis looks great, thanks See also this related tool: https://news.ycombinator.com/item?id=36907074
tikkun on HN
Objections
LLM auto-evaluators risk over-trust and circular scoring
From 2 comments
I like the support for Vector DBs and LLaMa-2. I'm curious as to whether and what influences compelled PromptTools, and how it differs from other tools in this space. For…
fatso784 on HNWhat does 'top 50%' responses mean here, though? You'd need to have a ground truth of how 'good' each score was to calculate that --and if you had ground truth, no need to use an…
fatso784 on HN
Requests
Add Qdrant support
From 1 comment
I'd like to see support for qdrant.
esafak on HN
Add OpenAI function calling support
From 1 comment
I like that it's not limited to single prompts and allows to have chat messages. It would be great if `OpenAIChatExperiment` could also handle OpenAI's function calling.
mmaia on HN
Provide a Dockerfile
From 1 comment
I'll put in a friendly request for a Dockerfile in the repo. I've been trying out AI tools as test cases for our supply chain security platform and had to cobble a Dockerfile…
robszumski on HN
Questions
How does it differ from other tools and what influenced it?
From 1 comment
I like the support for Vector DBs and LLaMa-2. I'm curious as to whether and what influences compelled PromptTools, and how it differs from other tools in this space. For…
fatso784 on HN
Where do thumbs up/down feedback values go and are they persisted?
From 1 comment
Similar tool I was about to look at: https://github.com/promptfoo/promptfoo I've seen this in both tools but I wasn't able to understand: In the screenshot with feedback, I see…
politelemon on HN
Alternatives named
ChainForge
From 1 comment
I like the support for Vector DBs and LLaMa-2. I'm curious as to whether and what influences compelled PromptTools, and how it differs from other tools in this space. For…
fatso784 on HN
promptfoo
From 1 comment
Similar tool I was about to look at: https://github.com/promptfoo/promptfoo I've seen this in both tools but I wasn't able to understand: In the screenshot with feedback, I see…
politelemon on HN