As of PHP 8.1, run-tests.php now caches SKIPIF evaluation by default. SKIPIF blocks are cached by a combination of their source code and test file directory. This is particularly problematic for PHPC since we use SKIPIF to drop collections before a test, and having run-tests.php defer to a cache entry means that code will not run before a test (inviting unexpected errors like duplicate key exceptions).

SKIPIF sections can opt out of caching by emitting "nocache" before any other output. In that case, run-tests.php circumvents caching, replaces the output with an empty string, and allows the test to run unskipped. There appears to be no way to out of caching and skip a test, which conceivably means that an unexpected skip (e.g. unexpected error dropping a collection) could lead to more tests being skipped than intended. My thread on php-internals goes into that in more detail, but at present there appears to be no workaround.

In any event, a basic solution to avoid caching-related test failures will be to have skip_if_not_clean conditionally emit "nocache" on success. And we'll need to gate that logic for PHP 8.1, since unexpected SKIPIF output could leak to borked tests in earlier PHP versions.