WiredTiger's oldest_id calculation doesn't include read only transaction snapshots (snap_min) and can lead to removing a historical record. If the older reader attempted to read the now removed record it would get WT_NOTFOUND.
The core file obtained from the customer on the linked related ticket indicates that this scenario is possible.
- reader session: snap_min (1208680567)
- oldest_id: (1208698455).
For example, if the key has the following updates in the update list
key1 -> U3(40) -> U2(20) -> U1(10)
Old reader (snap_min - 15, snap_max - 15)
oldest_id - 30
Before page eviction, the old reader can read the update U1. During the eviction, it is possible that due to the global visibility of U1 and U2 updates, U1 can get discarded. Later if the same old reader tries to read the key, it can get the WT_NOTFOUND error.
In another scenario, if the updates are already written to the history store. U1 update stop in is (20) and that is globally visible. The WT search logic of history store ignores the globally visible historical updates before checking whether it is visible to the current reader session.
Currently the oldest_id is used for two purposes.
- Reduce the pinned updates from the cache once they are stable
- Remove the historical versions as they no longer needed.