ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1026
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines - Transactions
    • Fully Compatible
    • ALL
    • v9.0, v8.3
    • 5
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description:

      Problem

      jstests/noPassthrough/txns_cache_errors/transaction_aborted_under_cache_pressure.js fails deterministically on the ThreadSanitizer variant (enterprise-rhel8-debug-tsan). The test sets CachePressureAbortSessionKillLimitPerBatch via a runtime setParameter, which races with the periodic rollbackUnderCachePressure thread reading the same global. TSAN aborts the server, the connection drops, and the test fails at load:

      Error: network error while attempting to run command 'setParameter' on host '127.0.0.1:20040' :: caused by :: Connection closed by peer
      failed to load: jstests/noPassthrough/txns_cache_errors/transaction_aborted_under_cache_pressure.js
      

      Reproduced on v8.3 patch 4213 (13/13 runs failed, ~9s each).

      Root cause

      gCachePressureAbortSessionKillLimitPerBatch is a plain, non-atomic int (src/mongo/db/session/kill_sessions.idl). The periodic job PeriodicThreadToRollbackUnderCachePressure reads it every cycle (int64_t killsTarget = gCachePressureAbortSessionKillLimitPerBatch; in periodic_runner_cache_pressure_rollback.cpp), so a runtime setParameter write races with that read:

      WARNING: ThreadSanitizer: data race
        Write of size 4 by thread T76 'conn1' (the setParameter):
          #0 storage_wrapper<int>::store   server_parameter_with_storage.h:244
          #3 CmdSet::run                    commands/parameters.cpp:493
        Previous read of size 4 by thread T71 'rollbac.ressure' (periodic job):
          #0 PeriodicThreadToRollbackUnderCachePressure::_init(...)::$_0::operator()
        Location is global 'mongo::gCachePressureAbortSessionKillLimitPerBatch'
      

      The race is latent on master and v9.0 as well; it surfaces on the 8.3 branch because burn_in_tags runs the changed test on the TSAN variant.

      Fix

      Set the parameter at mongod startup via the test's nodeOptions instead of a runtime setParameter, so no write occurs after the periodic thread starts:

      nodeOptions: {
          wiredTigerEngineConfigString: "cache_size=16M,cache_stuck_timeout_ms=600000",
          setParameter: {CachePressureAbortSessionKillLimitPerBatch: 1},
      },
      

      and remove the runtime db.adminCommand(

      {setParameter: 1, CachePressureAbortSessionKillLimitPerBatch: 1}

      ) block.

      Scope

      Test-only; no server changes.

      Verification

      Evergreen patch using burn_in_tags_gen (runs the affected test on all required and suggested variants, in repeated mode), including enterprise-rhel8-debug-tsan.

      Backports

      Required for v9.0 and v8.3 so the SERVER-132736 backports (BACKPORT-30842 / BACKPORT-30816) can pass the TSAN burn-in.

            Assignee:
            Ravi Giri
            Reporter:
            Ravi Giri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: