Name of Failure:
TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch
Link to task:
Context of when and why the failure occurred:
On 9.0+ sharded clusters, opening a change stream waits until the config server has caught up to mongos' cluster time (DRIVERS-3556 / SERVER-129623). Tests call mtest.AdvanceConfigClusterTime first, so the open doesn't block. This helper:
- Pings mongos and records its $clusterTime as the target.
- Opens a brand-new direct client to the config server. It has no cluster time to gossip.
- Runs appendOplogNote on the config server up to 100 times. Each note is one write, so it moves the config server's clock forward by one increment.
- If the config server's time is still below the target, an error is returned.
This creates a race. For example, what if the shard is already at {S, 1000}? Then appendOplogNote on the config server has a gap of 900 increments (being 100). In this case, you can get the error in the stack trace below.
Failure occurs when two things line up:
- The shard wrote more than 100 oplog entries in the current second before the helper ran. Most tests write a handful of documents, so the gap is small. A burst from a previous test, initCollection could create a large gap.
- The loop finishes before the wall clock reaches the next second. If the second rolls over partway through, the config server's next note lands in S+1 and passes the target.
Stack trace:
=== RUN TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch
cursor_test.go:640:
Error Trace: C:\data\mci\3edd\src\go.mongodb.org\mongo-driver\internal\integration\cursor_test.go:640
C:\data\mci\3edd\src\go.mongodb.org\mongo-driver\internal\integration\mongotest.go:234
Error: Received unexpected error:
config server cluster time is still behind mongos cluster time after 100 oplog notes
Test: TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch
Definition of done
Instead of looping up to 100 appendOplogNote calls, we should do the following:
- Take the $clusterTime document mongos returned from ping.
- Start a session on the config server client and load that time into it.
- Send one appendOplogNote in that session. The driver attaches the session's $clusterTime to the command.
- The reply's cluster time must now be at or past the target.
- Continue adding 5 margin notes in the same session to push it a little further past.
We should also gate mtest.AdvanceConfigClusterTime on 9.0+.