read_replica.js flaky: lastReadOplogLSN read once before the standby-read loop publishes it

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The disagg_storage integration test read_replica.js is flaky. It intermittently fails the final assertion that the read replica's lastReadOplogLSN equals the primary's oplog LSN, because the metric is read a single time before the replica's background standby-read loop has published it.

      uncaught exception: Error: expected NumberLong("7667194008422055970") to equal NumberLong(0)
        assert.eq@src/mongo/shell/assert.js:310:14
        @src/mongo/db/modules/atlas/jstests/disagg_storage/read_replica.js:111:8
      

      The assertion compared the primary's logServerManagerMetrics.oplogLSN
      (7667194008422055970) against the read replica's
      metrics.disagg.logServer.lastReadOplogLSN, which was still 0.

      Root cause. This seems to be only a test race, not a product bug. The read replica publishes lastReadOplogLSN from its background LSM-standby-read loop, asynchronously from when the replicated collection data first becomes visible. The test waits for hello.secondary, for the collection to exist, and for the document content (all of which pass), then reads lastReadOplogLSN exactly once.

      At the moment of the failing read (log ts 13:08:37.222), the standby-read loop had only just entered "Streaming oplog entries and checkpoint markers from log server" (_readLogStartLSN: ...969) and had not yet published a read LSN, so serverStatus() returned 0.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: