Picking which tests to run from the files a change touched

The game has 189 tests now, and 135 of them were added in one week. I had been choosing which ones to run after a change by typing words and running the tests with those words in their names. That missed three tests when the 58 meat monsters landed, so I wrote a script that reads the code and picks the tests a change can break. While I was getting the main branch passing again to try it, I also found a real bug in how rooms get furnished.

43 seconds, captioned: the store, a cell and the chief scientist’s room on floor 1, with furniture beside each monster.

Why was I choosing tests by word?

The whole suite takes about 25 minutes, so for a small change I only ran the tests whose names matched a word. That works until a change reaches a test whose name doesn’t say so. The meat monsters broke resistances, content_validator and furniture_in_site, and no word I’d have typed would have found them. The whole suite also hadn’t been run on the main branch since the night before.

This was the first of seven cleanup pieces I picked before starting another large feature. I’d asked for a proposal on what cleanup to do first, and I told it to adopt all seven in order.

How does the picker decide?

It’s one Python file, tools/test_picker.py, with the rule and a table in it. For each changed file it works in this order.

First, a table row. Each row is a pattern and what it picks. Some say the whole suite, for files nobody can reason about: project.godot, the test harness base, the terrain addon, and eight hub scripts like the party, the monster and the content database. An autoload’s script is the whole suite too, and the picker reads the autoload list from project.godot for that. Pages for people (docs/**, *.md) pick nothing. Content files pick the content tests, and anything under scripts/ picks at least the smoke test and the editor safety test.

Second, who names the file. The picker reads every script and scene once. A file names another if its text holds the other’s path, the other’s class_name, the folder it’s in (for a data file, at least two folders deep), or, for content, the file’s name in quotes. A test is picked if its script or scene names the changed file. tests/resistances.gd mentions MonsterProfile, so a change to the monster profile script picks it.

Third, if it can’t tell, it runs everything. That means a file no row matches and nothing names, a row that says so, or a pick of more than three quarters of the suite, since that many reasons is a hub.

I made the name search go only one step deep, and I measured why. A test’s script names 10 of the game’s 373 scripts at the median. With the scripts those name it’s 74, a step further it’s 188, and following everything reaches 347. At two steps, one ordinary script picks half the suite, so monster_profile.gd alone picked 96 of 189 tests and every landing would have been the whole suite. There’s a --deep flag if I want two steps anyway. The cost of one step is that a test two steps from a change can be missed, which is what the table’s hub and smoke rows are for. So the picker answers what’s least worth running, and the whole suite still runs before I stop for the day.

Here’s what it picks now:

A monster script, scripts/ai/meat_ai.gd: 6 of 189 tests.
A content file, content/loot.json: 11 of 189.
A hub file, scripts/party.gd: the whole suite.
Only docs/DESIGN.md and WORKLIST.md: 0.
The commit that landed the monsters: 107 of 189, with the three tests that were missed among them.

I hit one bug using it. When an editor test is running, the editor addon loads a copy of the terrain library named ~lib...dll, which git sees as a new untracked file in a folder with a whole-suite row. Every pick during an editor test came out as everything, until I taught the picker that a file starting with ~ isn’t a change. The self-test, tools/test_picker_test.py, builds a made-up project with git, then checks this checkout. It also fails if the table names a test that’s gone, so a renamed test can’t drop out quietly. It passes 59 checks in about 10 seconds.

Why did nine tests fail?

I had an agent trace each failure by running it alone and reading the old suite logs, which are kept for every worktree. Seven were fixed, one I left alone on purpose, and one wasn’t a bug.

The three furniture tests (furniture_in_site, prop_clutter, dungeon_dresser) and a real bug, which is the next section.
Two lighting tests (lights_up, failing_lights) followed the new surgery room. It’s 8 m high, so no spare lamp hangs in it (42 spares became 39), and its two lamps moved the total energy from 121.8 to 123.6. I changed the baselines.
laser_walls allows one warning now. The first cell’s Meat Sack is placed on purpose where it sees one square.
faces_in_game struck a Slab Warden with a blow of 1. The warden’s guard takes half of a blow, and half of 1 rounds to 1, so the whole blow was soaked and nothing happened. The test strikes with 10 now. A blow of 1 doing nothing to a guarded monster is a real bug, but a small one. I filed it and didn’t fix it.
turns measured the wall clock in two checks, so a busy machine’s stalls counted as game time (3.65 s where 1.7 was expected). Both now count the engine’s physics steps.
world_frames is left failing. Building the Breakout Site in one frame takes 593 ms against a 400 ms limit. I’ve already decided to spread that build over several frames, and I’m not raising the limit.

What was the dresser doing wrong?

The room dresser kept furniture off the nine squares around every pack of monsters. That was made for packs of several, whose members stand around the middle. Since the meat monsters, nearly every room has one monster standing alone, and the eight squares around each took a third of floor 1’s furniture. It fell from 523 pieces to 340, and three cells and the store were left bare.

A pack of one now keeps only its own square clear, and a pack of several still keeps all nine. Floor 1 has 523 pieces again. The film above shows the store with its pallets back beside its Meat Sack, a cell, and the chief scientist’s room with its desk, rug and pictures.

A grid of eight first-person views of dark industrial rooms with furniture, red meat growth on the ceiling and a health bar over each monster. — Eight stills from the film: the store, a cell, and the chief scientist’s room.

The tests also turned up a second dresser bug. Small things like a bucket or a jar could stand under a picture on the wall, and only the luck of the layout had been keeping them out. The dresser now passes the rugs and pictures to the code that places small things, which drops any spot under one.

Why did the first whole run look clean?

It looked nearly clean for a bad reason. That worktree had no island, because the island’s data isn’t in git and a new worktree doesn’t get a copy. So 19 tests printed that they were skipped, and the suite counted them as passed. The first run was 189 tests with 2 failed and 1 that passed only alone. I copied the island over from the main checkout (7.1 GB with the raw data) and ran again. That run was 189 tests with 1 failed (world_frames) and 0 that passed only alone. I filed that, so a new worktree gets the island and the suite counts skips.

What isn’t settled?

I haven’t decided if the dresser’s new rule is right, because a lone monster now keeps only its own square clear. For now it stands. I also haven’t used the picker on a partial pick through a whole landing, so I don’t know yet how it behaves there. The blow of 1 bug and the island fix are open, and so is a rare test failure after a load where one monster comes back with a full record when the save had it as built.