WT-11017 found an apparent out-of-order key while doing a key order check on the history store cursor. This was a false positive from the diagnostic check and the tests never exhibited any actual out-of-order keys in the tree. To hit this scenario the history store cursor would call next or prev and cross a page boundary. Other threads would simultaneously remove the lastkey seen and insert new key(s) at the edge (start or end) of the page. It’s then possible for the cursor to see a key that is less than the previous key on a next walk and vice versa for a prev call. 

This occurs due to the history store cursor using read uncommitted isolation. We should reassess the behaviour of the history store cursor and determine whether we should allow the cursor to see values that appear to be out of order.

WT-11017.pdfprovides an explanation for the above scenario.