Details
Description
SERVER-33126 added logic to prevent the stable timestamp from being set ahead of oplog visibility. It is a required piece for SERVER-29213.
`_advanceOldestTimestamp` was already doing this. However, `_advanceOldestTimestamp` early-returns if oplog visibility is behind the current oldest timestamp.
This combination creates a bug where SERVER-33126 passes an already pinned value to `_advanceOldestTimestamp`. The second pin attempt is unsuccessful, so the guard never executes.
Specifically how to hit this is significantly more involved; if oplog visibility wants to pin a write to the oldest timestamp, how come oplog visibility pinning didn't take effect on the previous write? Briefly, despite setting the oplog visibility to last applied coming out of initial sync, that update can be lost to a stale value.
In more detail: applying batches during initial sync sets the oldest timestamp to the last applied. Coming out of initial sync force sets the oldest timestamp to the last applied. Due to a race between multiple writers of this value, the oplog visibility thread can lose the force-set from coming out of initial sync and set it to a value from a, now-stale, read.