As part of the chunk cache project, we noticed some issues with using statistics as part of testing. Every single test seems to have some form of get_stat function, which is a lot of duplicated code. We could integrate this into wttest and make it even simpler to write tests.

But that only solves half of the problem - get_stat by itself tends to become racy. For example, in the chunk cache project, we would do a flush_tier and then assert something about the content of the chunk cache. But without some sort of delay, there's no guarantee that WiredTiger has updated the statistics before we open a stats cursor.

This is exacerbated a little by the development cycle - a local build is probably using debug mode, which will fetch the stats more slowly and hide races. Then the PR testing and commit queue testing also mostly use debug-mode tests, so the issue is unlikely to be spotted before it lands in develop and turns the waterfall red.

I would like to be able to replace code like this:

time.sleep(0.5)
stat = self.get_stat(...)
self.assertGreater(stat, 0)

with code like:

self.assertStatGreaterSoon(..., 0)

The strategy for waiting would need to be figured out - 0.5 seconds per call could add up. Perhaps it could repeatedly check in a loop until either 0.5 seconds have passed or there have been some number of retries.

The test_chunkcache set of tests would be a good place to start fixing this, since they've already got their own version of this.