upsert_write_conflict_retry.js in-transaction case is flaky when another operation consumes the injected write conflicts

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      The subtest an upsert inside a transaction surfaces the write conflict instead of retrying in jstests/noPassthrough/query/update/upsert_write_conflict_retry.js fails intermittently at roughly 0.3%. The upsert succeeds instead of returning WriteConflict, so the assertion fires:

      command worked when it should have failed: { "n" : 1, ..., "upserted" : [ { "index" : 0, ... } ], "ok" : 1 }
        at assert.commandFailedWithCode@src/mongo/shell/assert.js:1574
        at jstests/noPassthrough/query/update/upsert_write_conflict_retry.js:171:16
      

      Cause

      The test injects conflicts with the WTWriteConflictException failpoint using times: kWriteConflictCount (10). That failpoint is evaluated by a bare shouldFail() in src/mongo/db/storage/wiredtiger/wiredtiger_record_store.h with no predicate, so it is process-wide and fires for whichever operation reaches a write first. When other modifying operations on the node consume all 10 firings before the transaction's upsert performs its insert, the upsert never sees a conflict and returns ok:1.

      The test already anticipates this. The comment above the failpoint reads "Inject more than one conflict for margin against any other modifying op consuming a firing". The margin of 10 is not always enough on a busy inmem node.

      Observed

      Both runs on variant enterprise-rhel-8-64-bit-inmem-generated-by-burn-in-tags, task burn_in:no_passthrough_execution_control_with_prioritization_gen:

      • Patch 6a8f3f11de809e00071ed583: 2 failures / 560 repetitions
      • Patch 6a9087ac988593000716fd64: 1 failure / 576 repetitions

      Only burn_in surfaces it, because normal CI runs the test once.

      Scope

      The subtest was added by SERVER-66751 (commit 5d08617fd3f) and has not been modified since. SERVER-133957 edits the same file but does not touch this subtest; the rate is unchanged across both patches, one before and one after that change.

      Possible fixes

      • Assert on the failpoint's timesEntered after the command and skip the case when the firings were consumed elsewhere, rather than assuming the upsert got one.
      • Give WTWriteConflictException a predicate so the test can scope injection to the target namespace or thread. This requires a code change, since the current call site passes no predicate.
      • Raise the injection count, which reduces but does not remove the race.

            Assignee:
            Zixuan Zhuang
            Reporter:
            Zixuan Zhuang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: