ExportXMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Done
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: Evergreen
    • None
    • None
    • Go Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Name of Failure:

      TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch

      Link to task:

      https://parsley.corp.mongodb.com/test/mongo_go_driver_tests_windows_44_plus_zlib_zstd_support__version~4.4_os_ssl_40~windows_64_test_sharded_noauth_nossl_zstd_compression_3c254894

      Context of when and why the failure occurred:

      On 9.0+ sharded clusters, opening a change stream waits until the config server has caught up to mongos' cluster time (DRIVERS-3556 / SERVER-129623). Tests call mtest.AdvanceConfigClusterTime first, so the open doesn't block. This helper:

      • Pings mongos and records its $clusterTime as the target.
      • Opens a brand-new direct client to the config server. It has no cluster time to gossip.
      • Runs appendOplogNote on the config server up to 100 times. Each note is one write, so it moves the config server's clock forward by one increment.
      • If the config server's time is still below the target, an error is returned.

      This creates a race. For example, what if the shard is already at {S, 1000}? Then appendOplogNote on the config server has a gap of 900 increments (being 100). In this case, you can get the error in the stack trace below.

      Failure occurs when two things line up:

      • The shard wrote more than 100 oplog entries in the current second before the helper ran. Most tests write a handful of documents, so the gap is small. A burst from a previous test, initCollection could create a large gap.
      • The loop finishes before the wall clock reaches the next second. If the second rolls over partway through, the config server's next note lands in S+1 and passes the target.

      Stack trace:

      === RUN   TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch
          cursor_test.go:640: 
              	Error Trace:	C:\data\mci\3edd\src\go.mongodb.org\mongo-driver\internal\integration\cursor_test.go:640
              	            				C:\data\mci\3edd\src\go.mongodb.org\mongo-driver\internal\integration\mongotest.go:234
              	Error:      	Received unexpected error:
              	            	config server cluster time is still behind mongos cluster time after 100 oplog notes
              	Test:       	TestCursor_tailableAwaitData_ShortCircuitingGetMore/maxAwaitTime_less_than_operation_timeout/watch
      

      Definition of done

      Instead of looping up to 100 appendOplogNote calls, we should do the following:

      1. Take the $clusterTime document mongos returned from ping.
      2. Start a session on the config server client and load that time into it.
      3. Send one appendOplogNote in that session. The driver attaches the session's $clusterTime to the command.
      4. The reply's cluster time must now be at or past the target.
      5. Continue adding 5 margin notes in the same session to push it a little further past.

      We should also gate mtest.AdvanceConfigClusterTime on 9.0+.

            Assignee:
            Preston Vasquez
            Reporter:
            Preston Vasquez
            Matt Dale
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: