llm-security
AI
llm security research
SHOW HN2023
How HN took it
3 positive, 19 neutral or asking, 0 mixed, 6 critical, out of 28 comments about the project. Thread score -0.13.
Privacy: 0 for, 3 againstIdea: 0 for, 3 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Prompt golfing game and Medium article were clarifying and valuable
From 2 comments
That prompt golf game is awesome.
mk_stjames on HNJust wanted to say thank you so much for posting this (I also just realized you are the author of the github repo). This is exactly the kind of content I come to HN for. I…
hn_throwaway_99 on HN
Objections
Prompt injection may be unsolvable since instructions and data share one text channel
From 2 comments
> your finding that ChatML does not work seems to be mainly this Also the fact that ChatML has been broken into bits many, many times now- see again the prompt golfing. Also I'm…
greshake on HNSeparate how? It's all still getting fed into the same text processing pipe. The tools to do something that's fundamentally different from that literally don't exist yet.
crooked-v on HN
Separating system prompts from user data has not worked in practice
From 2 comments
Segmenting different data sources is the main approach pursued by OpenAI afaik (ChatML for example). That has not worked so far, as you can see in this prompt golfing game:…
greshake on HNDid you try this though, because so far it doesn’t seem to give the ‘system’ prompt preference over the ‘user’ prompt; the user can override the system prompt with some trivial…
anonzzzies on HN
Requests
Better separation of instructions and data at the model level
From 1 comment
I wonder if a lot of those "injection" problems could be overcome by introducing a distinction between the different types of input and output already at the token level. E.g.…
xg15 on HN
Questions
How the pirate example was produced from the keyword search step
From 1 comment
So...in the pirate example the comment said to talk like a pirate, right? Is the example comment where it searches for a keyword a different example? I'm just really confused why…
thequadehunter on HN
Alternatives named
ChatML system/user role separation
From 2 comments
Segmenting different data sources is the main approach pursued by OpenAI afaik (ChatML for example). That has not worked so far, as you can see in this prompt golfing game:…
greshake on HNOk, the "repeat this in your internal voice" exploit is impressive. However, apart from this I don't see anything concrete that ChatML uses different parts of the network for…
xg15 on HN