StarSkirmish Bench
AI
llm benchmark
SHOW HN2026
How HN took it
6 positive, 31 neutral or asking, 1 mixed, 10 critical, out of 48 comments about the project. Thread score -0.02.
Idea: 3 for, 5 againstPerformance: 2 for, 5 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting benchmark with time limits and a clean site
From 2 comments
This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks. It would be interesting to see the reverse, how long does each model take…
m_w_ on HNThe site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.
aleksiy123 on HN
Objections
Unclear what the benchmark actually measures
From 2 comments
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is…
iugtmkbdfil834 on HNYeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best…
chis on HN
Results may reflect memorized bot code from training data
From 3 comments
StarCraft probably has hundreds of bot implementations lying on GitHub. Which makes this bench a recall-from-the-training-set competition, essentially. I tried that method on old…
gritzko on HNYeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best…
chis on HNHow do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
jstummbillig on HN
Requests
Comparisons of harnesses and tools
From 2 comments
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers. ``` Anthropic (Claude models): Claude Code OpenAI (GPT models): Codex Google…
djoldman on HNI loved sc1 (and SC2) Do you plan on comparing harnesses also? Playing in pi, opencode Vs the original harnesses?
krzyk on HN
Results against stronger bots like Pluto
From 1 comment
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
BeaverGoose on HN
Questions
How do the agent-written bots compare to human bots and what can be learned from them
From 1 comment
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any…
cgearhart on HN
How do the best humans do against the LLMs
From 1 comment
How do the best humans do against the LLMs?
caughtinthought on HN
Alternatives named
Pluto StarCraft bot
From 2 comments
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
BeaverGoose on HNI would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and…
__cayenne__ on HN
BroodBench LLM StarCraft benchmark
From 1 comment
So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly -…
MichaelNolan on HN