Recently we added a scons cachedir validate mode, which checks that the md5 hash of the file in cache matches the file after its been pulled from the cache. This was to make sure cachefiles don't become corrupt while sitting in the cache.

However this check does protect that a cachefile was put in the cache incorrectly. The file itself may not be corrupt, but just incorrect. This mode would rebuild the file locally and then verify the local md5 hash matchs what is the cache for the given build signature. This would be similar to a clean build in terms of cost, so its intended to be used sparingly in a maintenance context.

This will also be useful for flushing out issues which will cause problems for reproducible builds. It is already known that certain windows builders will compile the same inputs but produce slightly different build outputs (https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html).