Testing
Tinderbox currently has 1,109 tests that are run just about every time I make a change. Because they’re run so often, it’s important that they run quickly. After speeding up two of the slowest tests, I took inventory; here are some observations.
- The whole suite runs in about 19sec, so the average test takes 17ms. Half the tests take less that 15msec.
- The slowest test right now tests the badge menu and takes 387msec.
- A rule of thumb holds that tests ought not to take more that 100msec. Thirteen tests miss that target.
- Four of the slow tests are slow because they involve the file system; for example, that slow badge menu test build a badge menu by looking for various directories of badge images.
- Two of the slow tests build an entire text window. That’s probably not appropriate for this kind of test.
- Several of the slow tests involve asynchronous tasks running on other threads. These tests fail if the job doesn’t get done on time and use a naive approach that waits the allotted time and checks whether the task was finished. These could be rewritten more intelligently.
- 7 of the 25 slowest tests involve a new technique that Tinderbox 7 will use to improve performance in Tinderbox maps.
- One of the slowest tests turned out to be an old relic that did nothing useful.