Factorio Learning Environment
AI
llm agent benchmark
SHOW HN2025
How HN took it
18 positive, 71 neutral or asking, 2 mixed, 7 critical, out of 98 comments about the project. Thread score +0.16.
Idea: 15 for, 6 againstDesign: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Better than sparse benchmarks because the game is unbounded
From 1 comment
Fantastic idea. It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume…
spieswl on HN
Objections
Largest-factory goal leads to sprawling belt loops; SPM would be a better metric
From 1 comment
Very cool and also pretty expected results tbh. Some thoughts: Factorio is a game that requires SIGNIFICANT amounts of thinking ahead, often requiring investments into things…
WJW on HN
Results were expected given Factorio's need for long-term planning
From 1 comment
Very cool and also pretty expected results tbh. Some thoughts: Factorio is a game that requires SIGNIFICANT amounts of thinking ahead, often requiring investments into things…
WJW on HN
Requests
Add tower-defence style tasks with biters
From 2 comments
Fantastic idea. It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume…
spieswl on HNOne thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps / seconds. The idea would be to test agents in…
noddybear on HN
Layout optimisation benchmarks
From 2 comments
Fantastic idea. It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume…
spieswl on HNOne thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps / seconds. The idea would be to test agents in…
noddybear on HN
Screenshot or multimodal input support
From 2 comments
OK, You’ve permanently nerd-baited me, and I wish to apply for a job at the Anthropic Factorio lab immediately. I can’t tell from the paper or these comments if you’re sending…
vessenes on HNDoes it provide screenshots of the game state? I, too, would struggle to play the game pretty effectively if I could not visually see the game.
kevmo314 on HN
Questions
Are images sent back to the agents?
From 3 comments
OK, You’ve permanently nerd-baited me, and I wish to apply for a job at the Anthropic Factorio lab immediately. I can’t tell from the paper or these comments if you’re sending…
vessenes on HNDoes it provide screenshots of the game state? I, too, would struggle to play the game pretty effectively if I could not visually see the game.
kevmo314 on HNAgents don't have access to screenshots, as we are purely evaluating text-only models. All reasoning is conducted over object representations of the game (with positions etc). I…
noddybear on HN
How does the LLM control the game loop?
From 1 comment
Everytime a paper like this comes out, I always have 1 question - How do they control the game using the LLMs? How does the control-feedback loop work? WHat tools, software and…
jharohit on HN
Is there a human-play benchmark?
From 1 comment
Is there a human-play benchmark (even informally) for this style of interface? Not saying it's necessary or even relevant, I'm just curious to know what programmatic Factorio…
mNovak on HN
Alternatives named
Pokemon Red reinforcement learning agent
From 1 comment
There was a HN post here not too long ago about a team that used reinforcement learning to train an agent to beat pokemon red. They mentioned how they had to tweak the cost…
scottmsul on HN
Terraform Factorio provider
From 1 comment
Someone created this Terraform provider for Factorio a few years ago: https://registry.terraform.io/providers/efokschaner/factorio/latest/docs I think this could be a good…
noddybear on HN