upsert_write_conflict_retry.js retry-limit case is flaky because background writers drain the counted WTWriteConflictException budget

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1022
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      jstests/noPassthrough/query/update/upsert_write_conflict_retry.js failed in the no_passthrough_execution_control_with_prioritization suite (enterprise-rhel-8-64-bit-inmem,
      commit 5961b28497c):

      ✘ upsert insert write-conflict retry > an upsert insert that exceeds the retry limit fails with WriteConflictRetryLimitExceeded
      command worked when it should have failed: { "n" : 1, ..., "upserted" : [ ... ], "ok" : 1 }
      

      The subtest sets internalQueryWriteConflictRetryLimitMax: 5, arms the WTWriteConflictException failpoint with {times: 50}, and expects the upsert to abort with WriteConflictRetryLimitExceeded after 6 consecutive conflicts. Instead the upsert succeeded.

      Root cause

      WTWriteConflictException is a global counted failpoint: WT_OP_CHECK fires it on every WT modifying cursor op on the node, with no namespace filtering (src/mongo/db/storage/wiredtiger/wiredtiger_record_store.h). The task log shows the JournalFlusher background thread hitting a WCE on its upsertOplogTruncateAfterPointDocument write while the failpoint was armed:

      "ctx":"JournalFlusher","msg":"Caught WriteConflictException",
      "attr":{"operation":"upsertOplogTruncateAfterPointDocument", ...}
      

      Its internal writeConflictRetry loop spins with minimal backoff, so it drained the 50-firing budget in ~85ms (failpoint timesEntered went 20 -> 70). The user upsert therefore never accumulated kLimit + 1 = 6 consecutive conflicts before the budget ran out; the insert then succeeded and assert.commandFailedWithCode failed. The test already disables the periodic noop writer, but the JournalFlusher (and other background writers, e.g. the sessions-collection refresh) cannot be similarly disabled.

      The same fragility exists in the in-transaction subtest ({times: 10}): if background writers drain the budget before the transaction's update performs its write, the update succeeds and the expected WriteConflict error never surfaces.

      The race is timing-dependent, so it can strike in any suite that runs this test; the execution-control-with-prioritization variant merely altered timing enough to expose it.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: