Codex-Minecraft-Gameplay Review 2026 — A Bounded Windows Input Toolkit That Teaches Computer-Use Agents to Actually Play Minecraft
✅ Pros
- • Treats computer use as a real engineering problem instead of a demo: every input is bounded (holds 0.02–5 s, settle 0–2 s, relative mouse totals within ±2000), every action is captured before and after, and the adapter returns structured exit codes — 0 success, 1 failure, 2 refused-or-unknown, where 2 never counts as success — so the agent cannot fool itself into thinking a no-op worked
- • Physical scan codes, not virtual-key abstractions: the key map uses physical PC scan codes (w = 0x11, space = 0x39, with extended keys flagged separately) and the README is explicit that names assume matching in-game bindings and the adapter never remaps the user's controls — honest about the layer it operates at
- • Verification is built into the action loop: the skill instructs the agent to classify gameplay versus menus from a fresh capture, establish dimensions, footing, selected item and resources before acting, inspect the result after every action, and use checked sequences with expect_before at menu transitions, watch for stable conditions during holds, and progress checks when repeating actions
- • The sequence runner turns 'did it work?' into a JSON plan with pixel checks: a plan declares a client size, reference image, comparison regions and a max_mean_error threshold, and minecraft_sequence.py validates the plan (--dry-run sends no game input), executes the bounded batch, and writes a report — with rules like 'pixel changes are stall signals, not a semantic count of movement or collected items'
- • Honest recovery semantics: a stopped sequence may have partially moved the player or consumed items, so the agent must continue from a fresh observation rather than replay a resource-consuming prefix, and the docs say to distinguish completed input, passed comparisons and the user's actual gameplay objective when reporting
- • Clean license and real tests: Apache-2.0 with an explicit NOTICE, Python 3.8+ standard-library-plus-Pillow runtime, and a unittest suite (test_minecraft_control.py, test_minecraft_sequence.py) that runs against fake inputs and synthetic images so the mechanics are verified without a game
⚠️ Cons
- • Windows-only by design: the toolkit drives a local interactive Windows desktop with Win32 input events, so macOS and Linux agents are out, and the process executing the adapter must be able to access the game's desktop — private desktops or incompatible privilege levels can break enumeration, input or capture in ways the docs can only describe, not fix
- • Brings the game, not the brains: the repo explicitly does not include an agent model, a game client, a world save or a game-memory interface — it is a control and observation layer for an agent that already exists, which means the quality of play depends entirely on the computer-use model in front of it
- • Gameplay depth is limited by design: individual actions do not inspect inventory, hazards or item identity; menu coordinates must be read from the current screenshot (the adapter refuses to guess); camera control is relative deltas that need calibration; and the runner enforces 32-step, 30-second and 5-second-per-hold limits — fine for gathering and crafting demos, not for autonomous long-horizon play
- • Young and single-purpose: created September 6, 2026 with a small contributor surface, one documented X post as the main signal, and no issue tracker history at review time — the interesting validation (how far a computer-use agent can actually get in survival mode) is still anecdotal
- • The screenshot observation layer has classic computer-use blind spots: overlapping windows can appear in captured pixels, minimized windows must be restored manually, and a resize changes the mapping between displayed points and native client pixels — the docs give the conversion formula (native = displayed × source / image) but the agent has to apply it correctly every time
Developers who want a safe, verifiable harness for benchmarking or demonstrating computer-use agents on a real interactive application — Minecraft is the sandbox, but the value is a bounded Windows input adapter with checked sequences, an F8 kill switch and honest exit codes that can be reused as a template for letting agents drive any foreground desktop app without letting them run wild
Free and open source (Apache-2.0). Clone the repository and open it as a local Codex project on the Windows PC running Minecraft; create a virtual environment in the skill's runtime directory (py -3 -m venv .venv), pip install the requirements (Python 3.8+, Pillow) and run the unittest suite once. Requires a Windows desktop with Minecraft running in a visible window, plus a computer-use agent (Codex with a vision-capable model, per the author's demo). No API keys, no cloud, no per-seat cost.
The Problem: Computer-Use Demos That Lie
Watch a computer-use agent demo and you’ll notice a pattern: the agent says it did things — mined a block, opened a chest, walked somewhere — and the video cuts away at exactly the moments that would prove it. The gap between “the model believes it acted” and “the action actually happened on screen” is the whole game, and most demos paper over it. wz1119’s Codex-Minecraft-Gameplay, created September 6, 2026, takes the opposite approach: a Windows keyboard, mouse and screenshot toolkit where the agent observes the game, chooses actions, and verifies the result — and every input is bounded by hard limits so a hallucinating model can’t hold a key down forever or fling the mouse across the screen. In two days it passed 120 stars on an Apache-2.0 license.
The scope is deliberately narrow and honest: the toolkit does not include an agent model, a game client, a world save or a game-memory interface. It is the hands and eyes — the Python runtime provides controls and optional visual checks, and Codex (or another computer-use agent) provides the decisions. Minecraft is the chosen proving ground because it is a real, visible, interactive application with immediate visual feedback: move, look around, gather materials, craft, explore, build — all through the visible game interface, the way a human plays.
The Bounded Input Adapter: Guardrails as Architecture
minecraft_control.py is the core: a bounded Windows input adapter for a foreground Minecraft window. Its guardrails are the architecture, not an afterthought. Key holds run between 0.02 and 5 seconds (MAX_SECONDS = 5.0); a --settle window (0–2 s, default 0.15) waits after key release before capture; relative mouse totals are integers between −2000 and 2000; motion is distributed over the hold, so changing the view and holding attack at the same time sweeps across targets. F8 is the interrupt key, and a persistent stop file (.minecraft-control-stop) interrupts input — a belt-and-suspenders kill switch for when the model goes off the rails.
The key map is built from physical PC scan codes with extended keys marked separately — w is scan code 0x11, space 0x39, arrow keys are extended — rather than virtual-key codes, which keeps the adapter honest about the layer it operates at. Supported keys cover movement, jumping, sneaking and the hotbar (w a s d e q f t h space shift ctrl esc enter tab f3 f5 arrows 1–9); buttons are left, right and middle. The README is explicit that names assume matching in-game bindings and the adapter never remaps the user’s controls. Windows are found by title and process executable; if more than one matches, the agent adds --hwnd from the current status output, and handles change when the game restarts. Captures default to a max width of 1600 but support --max-width 7680 to preserve native dimensions for sequence references; the output carries source_size (native client dimensions), image_size (saved dimensions) and the file path, so the agent can convert displayed menu points back to native pixels.
Checked Sequences: Making ‘Did It Work?’ a JSON Plan
Where the toolkit gets interesting is minecraft_sequence.py, a sequence runner that validates a JSON plan, loads its reference images, and executes a bounded series of inputs with local visual comparisons. A plan declares a client_size, a max_seconds budget, named checks — each with a reference image, comparison regions, a mode (pixels) and a max_mean_error threshold — plus a start_check and an ordered list of steps, each step able to demand expect_before and expect_after states. The docs walk through an observation-only template whose step sends no input at all: it just confirms a stable region stays stable, proving the file format before any action is added.
The runner validates the plan with --dry-run (no game input, no live inspection), then executes with --focus, writing a capture and a JSON report. The skill layer adds the strategy on top: choose a milestone from the goal and observed resources; separate camera alignment from sustained mining (relative deltas are spread through the hold); verify clearance before advancing; inspect ingredients, recipe layout, quantities, cursor contents and destination slots before crafting, then verify the output in inventory afterward — because “a stable panel outline confirms menu layout, not item identity or crafting success.” The sequence docs add the memorable rule: pixel changes are stall signals, not a semantic count of movement or collected items.
Recovery, Interrupts and Honest Reporting
The skill’s inspect-and-recover rules are where the computer-use philosophy shows. A stopped sequence can have partially moved the player, moved ingredients or consumed items — so the agent must continue from a fresh observation and never automatically replay a resource-consuming prefix. When movement stalls, the agent inspects body alignment and reachable space before extending holds or mining more blocks. There is one live input owner; F8, focus loss and the stop file all interrupt input; a user interruption is treated as a stop, resumed only when requested. The skill tells the agent to pause singleplayer through its visible menu when appropriate and to verify the requested outcome before reporting completion — distinguishing completed input, passed comparisons and the user’s actual objective. That last line is the anti-demo discipline in one sentence: don’t report “done” because you sent the keys; report it because the screen changed the way the plan said it would.
Honest Boundaries and Who It’s For
The boundaries are stated plainly: Windows-only (the adapter drives a local interactive desktop), no agent model or memory included, gameplay depth capped by the 32-step / 30-second / 5-second-per-hold runner limits, and the classic computer-use blind spots documented rather than hidden — overlapping windows can appear in captures, minimized windows need manual restore, and resized captures require converting displayed points back to native coordinates. Created two days before this review with a small contributor surface and one X post as the main signal, it is young and unproven at long-horizon play.
But as a template for safe, verifiable computer use, it’s valuable beyond Minecraft: a bounded input adapter with a physical key map, an F8 kill switch, structured exit codes where refusal is never success, and a JSON sequence format that checks pixels before claiming victory. For developers who want to benchmark computer-use agents against a real interactive application — or who just want to watch Codex gather wood without setting the machine on fire — this is the most disciplined harness yet.