Auditing a day of agent work and building tooling from it

I measured where the first day of building Blobber went and turned the findings into tooling: one shell script that runs Godot, a base script that every test extends, a film recorder on top of Godot’s movie writer, and the files my agents read before they start. That evening eleven branches landed through it, built by three agents at a time.

Blobber is a first-person party RPG in Godot 4.7 that I started the evening before. I build it by directing AI coding agents: a main session I talk to, and background agents (workers) that each build one feature in their own git worktree.

Where did the first day’s effort go?

An agent read the transcripts of the main session and of the 15 workers run so far, and counted. These are its counts. I have not checked them.

The biggest cost was the main session: one conversation of about 822K tokens, re-read on every turn. The fix is a fresh session after each landed feature, and that only works if everything the session knows is in files.

Running Godot through one script

tools/gd.sh is now the only way Godot gets run, by me or by an agent. These are four of its five modes, from the script’s header (movie is further down):

#   tools/gd.sh import [-C dir]                  re-import (new class_name scripts, assets, after a merge)
#   tools/gd.sh check  [-C dir]                  import, then load the game for 90 frames
#   tools/gd.sh run    [-C dir] <scene> [-- a b] run a scene headless (logic tests)
#   tools/gd.sh shots  [-C dir] <scene> [-- a b] run a scene in a window (screenshots only)

-C dir picks the worktree, so the copy in the main checkout can run any worker’s branch.

Every mode goes through the same two functions in tools/gd.sh. The first runs the console build of Godot under GNU timeout and keeps the output and the exit code. The second decides whether the run failed:

godot() {
	local raw
	raw="$(timeout "$LIMIT" "$GODOT" --path "$dir" "$@" 2>&1)"
	code=$?
	out="$(printf '%s\n' "$raw" | filter)"
}

report() {
	local label="$1" failed=0
	[ -n "$out" ] && printf '%s\n' "$out"
	if [ "$code" -eq 124 ]; then
		echo "gd.sh: $label TIMED OUT after ${LIMIT}s (a script error in a harness leaves Godot waiting; look above)"
		failed=1
	elif [ "$code" -ne 0 ]; then
		echo "gd.sh: $label exited with code $code"
		failed=1
	fi
	if printf '%s\n' "$out" | grep -q "SCRIPT ERROR"; then
		echo "gd.sh: $label printed a SCRIPT ERROR"
		failed=1
	fi
	return $failed
}

A run fails three ways. timeout returns 124 when the limit passes (150 seconds unless GD_TIMEOUT says otherwise). The scene quits with a non-zero code. Or the text SCRIPT ERROR appears anywhere in the output, even from code the test wasn’t about. The timeout is there because a test script that fails to parse never reaches its own quit call, and Godot then waits forever.

import is --headless --import, printing only lines that hold ERROR, WARNING or an at: location. It has to run after a merge or after adding a script with a class_name, because until then the new class doesn’t resolve. check is the import and then --headless --quit-after 90. run is --headless <scene> -- <args>. shots is the same in a 1280x720 window, because a headless run draws nothing.

Godot prints the same harmless lines on every headless run, so filter drops them with awk before anything is judged. skip makes a dropped error take its indented at: line along:

filter() {
	awk '
		/^Godot Engine v/ || /^Vulkan / || /^OpenGL / || /^[[:space:]]*$/ { next }
		/^\[ *[0-9]+% \]/ || /^\[ DONE \]/ || /\[0m$/ && /(reimport|scan|first_scan|update_scripts|loading_editor)/ { next }
		/ObjectDB instances were leaked at exit/ ||
		/resources still in use at exit/ ||
		/Parameter "material" is null/ ||
		/BUG: Unreferenced static string/ ||
		/Pages in use exist at exit in PagedAllocator/ ||
		/A Thread object is being destroyed without its completion/ ||
		/Please call wait_to_finish\(\)/ { skip = 1; next }
		skip && /^[[:space:]]+at: / { next }
		{ skip = 0; print }
	'
}

Keeping the tests

Tests now stay in tests/. Each is a .tscn holding one Node with a script, and the script extends tests/harness_base.gd and overrides one function. This is most of tests/smoke.gd, the first kept test:

func scenario() -> void:
	await boot()
	check_eq(party.members.size(), 4, "a party of four")
	check(not party.groggy, "the party is not groggy once it has members")
	check(party.input_enabled, "the party can move")
	for member in party.members:
		check(member.is_alive(), "%s is alive" % member.name)
		check(member.action_damage(PartyMember.MELEE) > 0, "%s can hit" % member.name)
		check(not member.known_skills().is_empty(), "%s knows some skills" % member.name)
	check_eq(party.inventory.items.size(), 8, "the party starts with its eight rations and nothing else")
	check(get_tree().get_nodes_in_group("monsters").size() > 30, "the world has its monsters")

The base script’s _ready() awaits scenario() and then calls finish(), which prints checks: N passed, M failed and quits Godot with the number of failures as the exit code, capped at 100. gd.sh turns a non-zero exit into a failed run, so nothing parses test output.

boot() stands up the real game, the same scene the player gets:

func boot(with_intro := false) -> void:
	load(LEVEL_SCRIPT).skip_intro = not with_intro
	room = load(SCENE).instantiate()
	add_child(room)
	party = room.get_node("Party")
	if with_intro:
		await wait(6.5)
	else:
		await frames(3)

skip_intro is a static var on the level’s script, so it can be set before the scene exists. When it is set, the level’s _ready() skips the wake-up and calls a short function that powers the Meat Machine, marks it used, and builds the default party of four from the creation screen’s defaults. A test is ready after 3 frames. The old scripts waited 6.5 seconds for the opening and 4 more after confirming the party. The session reported a whole test run at about 1.7 seconds.

Input goes in the way a player’s does. key() builds an InputEventKey press and release and hands them to Input.parse_input_event(), so the game’s own handlers see them. hold() wraps Input.action_press() and Input.action_release() round a wait. A test can also call the game directly, as in monster.take_damage(n).

Logic is always judged headless, because a windowed run receives my real keyboard and mouse.

The suite is a loop, one Godot process per scene (from tests/README.md):

for scene in tests/*.tscn; do tools/gd.sh run "$scene" || echo "FAILED: $scene"; done

There was 1 scene with 19 checks in the afternoon and 37 scenes by the end of the day.

Filming a feature with Godot’s movie writer

Every feature the player can see or hear now gets a 20 to 60 second captioned film, and I get the film before I’m asked whether to merge.

I wanted to see the game in action while I’m away from the machine. Until then I only got screenshots.

tools/gd.sh movie runs a scene with Godot’s movie writer turned on:

godot --resolution 1280x720 --write-movie "$avi" --fixed-fps 30 --disable-vsync "$scene" -- "$@"

With --fixed-fps 30 the engine steps the game by exactly one thirtieth of a second per recorded frame, no matter how long the frame takes to draw. A wait(6.0) in a scenario is 6 seconds of film on any machine.

A film scenario is a test with captions, kept in films/<feature>.gd so the film can be recorded again after a change. This is the first half of films/tour.gd:

func scenario() -> void:
	await boot()
	caption("The vat bay")
	await wait(1.0)
	await pan(360.0, 6.0)
	await hold(&"move_forward", 2.0)

	var yield_monster := monsters_named("Spoiled Yield")[0]
	var target := yield_monster.global_position + Vector3.UP
	place(Vector3(-3.0, party.position.y, -16.0))
	face(target)
	caption("The yard: every recovered member acts while attack is held")
	await hold(&"attack", 10.0)

caption() is a Label on a CanvasLayer at layer 100, in 24 px type with an 8 px black outline. pan() is a Tween on the party’s rotation_degrees:y. place() teleports the party and face() turns it to a point in the world.

The movie writer leaves an .avi. tools/movie.py encodes it to an .mp4 that plays on a phone, then deletes the .avi:

    command = [
        imageio_ffmpeg.get_ffmpeg_exe(), "-y", "-loglevel", "error", "-i", source,
        "-c:v", "libx264", "-preset", "medium", "-crf", "27", "-pix_fmt", "yuv420p",
        "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart", target,
    ]

There was no ffmpeg on the machine. The Python package imageio-ffmpeg carries one, and get_ffmpeg_exe() returns its path. The 38.8 second tour came out at 6.4 MB.

An agent can’t watch a video, so tools/frames.py makes the picture above. It reads the film’s length from ffmpeg, takes 8 frames at the middle of 8 equal slices, scales each to 640 px wide and pastes them 2 across with Pillow. Whoever makes a film reads that sheet and checks that each caption’s shot shows what the caption says. The first take of the tour spent ten seconds facing a wall, and the sheet caught it.

gd.sh first looked for movie.py under $dir, the worktree being filmed with -C. Fifteen minutes later I had that changed to $(dirname "$0"), the script’s own folder, so a worktree cut before the encoder existed can still be filmed.

Recording opens a real window, so anything I type at the machine can leak into the film.

Writing down what an agent needs before it starts

Every session and worker now starts from CLAUDE.md (63 lines when written) and docs/ARCHITECTURE.md (166 lines), so a brief is only the task.

CLAUDE.md opens by pointing at four files: the architecture map, WORKLIST.md (what is open), docs/DESIGN.md (what I have decided) and docs/HANDOFF.md (what is in flight, rewritten before a session ends). Then come the traps of the machine. The heredoc one reads “large heredocs fail here and \\ is silently collapsed to \”, and the rule is to create files with the Write tool.

A worker doesn’t commit, merge, push, switch branch or spawn agents, and leaves its work uncommitted in its worktree. It decides small open questions itself and lists them. It keeps a running note outside the repository (done, left to do, verified), so that a worker cut off by a usage limit is resumed from the note and not started again. And it reports in a fixed shape (from CLAUDE.md):

- Report in at most about 6 KB, under exactly these headings: **Built** (what the player
  gets, with numbers); **Decisions I made**; **Files** (added, changed); **Verified** (what
  was run, counts passed and failed); **Film** (the .mp4's full path and what it shows, in
  order); **Not verified**; **Needs the user's answer**;
  **Worklist** (items closed, changed or added).

The architecture map has a section per system (which script owns it, what other code calls) and a table for the files every feature touches. By the audit’s count all 4 merge conflicts were two branches adding lines at the same spot in one of those. Two rows:

| `party.gd` `BINDINGS` | One line per new key action, kept in the dictionary's order of appearance. New state: one commented `var` beside the related ones. New signal: beside the other signals. |
| `ai/monster_profile.gd` | One static function per monster type, added after the last. |

A brief now has four parts: where to work, my request word for word with what it means as a numbered list, the few files to read first, and which files the worker owns with what its test and its film must show.

Answering the work list from my phone

WORKLIST.md holds everything open in the game, one checkbox line per item, and the same items are on a private web page where I type answers under them.

The file has six sections, and the section gives the id its letter: Q a decision waiting on me, S a system not built, H a stub, N a placeholder number, R a rough edge, K housekeeping. Ids are never renumbered, because the page and the design record refer to an item by its id. This is one question before and after my answer:

- [ ] **Q4 Loading-zone markers.** The two zones (bay corridor, exit hall mouth) draw an amber floor patch and a "LOADING ZONE" label. Keep visible?

- [x] **Q4 Loading-zone markers.** The two zones (bay corridor, exit hall mouth) draw an amber floor patch and a "LOADING ZONE" label. Keep visible? Decided: no visible markers. To build: S25.
- [ ] **S25 Remove loading-zone markers.** Decided (Q4), not built: take out the amber floor patch and the "LOADING ZONE" label.

A question says what the game does today, so I can answer it without opening the game. An answer ticks the Q, goes into docs/DESIGN.md, and becomes a new S item if it needs building. The list had 91 items when it was written and 231 by the day’s last commit.

The page is a Claude artifact with a small database behind it: one row per item, with the same ids. The main session reads it at the start of a session and before every landing, looking for rows with an answer that aren’t marked done. It acts on each and marks it done with a note of what happened.

My first 27 answers settled 15 questions with no work and made 12 build items, which went to two workers. Each added new items to the list in its own worktree. Both worktrees were cut from the same commit, so both took the same next ids, and one set had to be renumbered before landing. The rule since then: workers don’t edit WORKLIST.md, docs/DESIGN.md or tests/README.md. They report the changes and the main session applies them at landing.

After 38 more answers I set a standing rule for the evening: no more than three workers at a time, merge what passes the kept tests on the merged result, and start nothing that depends on work that hasn’t landed. The queue is a list in docs/HANDOFF.md, each line a branch, its items, and what it waits for:

- `resistances`: S38, S39, S40. After `weapons` (same party and item files).
- `save`: S4. After wave 1 (it must cover the event log, kill facts, new item fields). Put
  serialisation in its own scripts so later workers add their state to it.
- `builder-2`: S41 NPC editor, S47 drafts and published. After `content-core`.
- `spells`: S9, spell book. After `resistances` (damage types) and `weapons`.

The three branches of the first wave each got a list of the files they own, so they could run at once. Eleven branches landed that evening.

Also done

• tools/land.sh <branch> <message-file>: commits a worker’s worktree, merges it with git merge --no-ff, runs tools/gd.sh check on the result, and fast-forwards every other clean worktree.
• A quest builder inside Godot, with forms generated from the content schema: 16 kinds of condition and 28 kinds of effect.
• Character creation with point buy: 25 points, no stat above 25 or below 3.
• Three tiers of chests, a flying monster, 26 skills and XP from kills.
• Resting, which costs one food per party member, and surgery on organs, paid for in XP.
• A town, a sunken dungeon, a camp and a tower outdoors.
• The calendar runs at ten times play speed, so a full day and night takes 2 hours 24 minutes of play.
• From the evening run: save and load, 24 spells in three schools, a minimap and a map with the player’s own notes, twelve damage types.