The build system currently defines targets which execute unit tests. However, these executions are unconditional: the test always runs.

The build system, however, knows whether the test binary is up to date. Under the assumption of deterministic behavior in the tests, it should be the case that if none of the dependencies of a test have changed, then the outcome of that test should not have changed. So we can use the build system's knowledge of the up-to-date-ness of the test binary to avoid re-running tests for which no dependencies have changed. The unit test binaries are a particularly appealing first step for such work.

To implement such a feature we can emit the output of the test to a log file, and the result of the test to a "status" file, and then add new per-unit-test aliases that depend on these output files. When the alias is run, if the test has not yet been built or run, it will be built and run, with the output and result captured to the file. An immediate request to re-run the test will observe that the outcome files are up to date and not re-run the test. We can then aggregate all of the aliases for test status into one master test execution target. Running that master target will first build and run all of the unit tests. Subsequent runs will be instant, since the targets will be up to date. Later, after a developer makes a code change, re-running the master target will only re-run those unit tests for which the code change would cause them to recompile.

Potentially, we could use this technique in the CI loop to, to even greater advantage. There, the SCons caching can actually cache the results of running unit tests, so we can avoid re-running tests for which we have a cached status. This would mean, for instance, that we could re-use cached test outcomes from the commit before a commit which was subsequently reverted, avoiding the need to run the test binaries at all.

This technique could potentially be extended out to integration and JS tests, but there are challenges. We would first need to teach the build system about all of the JS tests and associated inputs like suite definitions. This would probably require redefining parts of resmoke to be a re-usable library which SCons could invoke to discover these components. We would also need all JS tests to explicitly identify the binaries that they needed to run, so that we could correctly express the dependencies. We would also likely need to write a JS scanner to identify usage of common library code. However, the gains would be huge. A commit which altered only a few JS tests or JS support files could be filtered down to running exactly the subset of tests that were affected. A commit which altered only mongos could run only those JS tests which actually used mongos.

This technique cannot be used for tests that are intentionally non-deterministic, like fuzzers.

A PoC of this feature was demonstrated at the dev prod demo day in early May. The PoC code (which still needs a lot of work) is available here.