network_error_and_txn_override.js leaks an open wrapped txn when a test ends inside a CRUD op (mochalite)

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Integration
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      jstests/libs/override_methods/network_error_and_txn_override.js with wrapCRUDinTransactions:true only commits the currently-buffered wrapped transaction when a subsequent non-txn-compatible command arrives (see setupTransactionCommand, lines ~636--664: the else branch commits only when ops.length > 0 AND a non-CRUD command comes next). There is no end-of-test / shell-teardown hook that flushes a still-open wrapped txn.

      For mochalite tests (see jstests/libs/mochalite.js) whose last it() body ends with a wrapped CRUD op, the final transaction is therefore never committed. The abandoned txn remains active on the primary for transactionLifetimeLimitSeconds (default 86400 in the multi-stmt-txn suites), and CheckReplOplogs / CheckReplDBHash then hang on the following fixture — observed as a task-timed-out with hang analyzer SIGKILL.

      Concrete evidence

      Reproduced locally on v8.2-staging with replica_sets_multi_stmt_txn_jscore_passthrough + jstests/core/index/geo/geo_s2stale_polygon_parse.js (3 mochalite iterations):

      • Primary log shows New transaction started for txnNumber:0, :1, :2; only txnNumber:0 and :1 receive a Received commitTransaction. :2 is started, its insert/find succeed, and then no commit ever arrives.
      • Immediately after, CheckReplOplogs freezes the secondary, fsync-locks the primary, and hangs indefinitely; WT stable timestamp is pinned at the last committed opTime and never advances.
      • Test-shell side, the last override log line is the response to the third-iteration find. There is no setupTransactionCommand :: Committing transaction NumberLong(2) following it, matching the flow: after mochalite's final it() body returns, no further server command is issued, so the override's commit trigger never fires.

      Analogous production symptoms in the sharded/replset multi-stmt-txn passthroughs on the same test:

      • BF-44810 (multi_shard_multi_stmt_txn_jscore_passthrough / others, CheckReplDBHash timeout, {{CMD fsync {lock:true}

        }} in the log)

      • BF-44833 (replica_sets_multi_stmt_txn_jscore_passthrough, CheckReplOplogs timeout, mongod SIGKILL from hang analyzer)

      Prior art / related:

      • SERVER-128348 — same override, same class of bug (txnRunCommandOverride retaining dangling txn state across mocha-style hooks in implicit_change_stream_v2.js).
      • BF-43660 — production symptom of the same shape on disagg_change_streams_multi_stmt_txn_mongos_passthrough: commitTransaction@network_error_and_txn_override.js:538 frames with "Transaction was aborted / NoSuchTransaction."
      • SERVER-108224 — canonical "mochalite bypasses passthrough runners" (fixed, runner-level only, does not touch txn flushing).
      • BF-38850 — earlier mochalite × override interaction closed with a revert.

      Proposed fix

      Install an end-of-test flush in network_error_and_txn_override.js that commits the currently-buffered wrapped txn if one is still open. Two triggers, so it covers both mochalite and non-mochalite tests:

      1. Wrap globalThis.__mochalite_closer at assignment time so that after mochalite's runTests returns, commitTransaction(lastConn, lastLsid, txnOptions.txnNumber) is invoked if ops.length > 0. The override never imports mochalite — intercepting the global assignment sidesteps the import boundary cleanly.
      2. Optionally, register a mongo-shell exit hook (if a suitable API exists) as a belt-and-suspenders path for non-mochalite tests that finish inside a wrapped txn.

      Sketch:

      function flushPendingWrappedTxn() {
          if (!TestData.networkErrorAndTxnOverrideConfig?.wrapCRUDinTransactions) return;
          if (ops.length === 0) return;
          logMsgFull('flushPendingWrappedTxn',
                     `Committing dangling txn ${txnOptions.txnNumber} on session ` +
                     `${tojsononeline(lastTxnLsid)} at test teardown`);
          try { commitTransaction(lastTxnConn, lastTxnLsid, txnOptions.txnNumber); }
          catch (e) {
              jsTest.log.info("flushPendingWrappedTxn: commit failed; aborting", {error: e});
              try { abortTransaction(lastTxnConn, lastTxnLsid, txnOptions.txnNumber); } catch (_) {}
          }
      }
      
      let _closer;
      Object.defineProperty(globalThis, '__mochalite_closer', {
          configurable: true,
          get() { return _closer; },
          set(fn) { _closer = async () => { try { await fn(); } finally { flushPendingWrappedTxn(); } }; },
      });
      

      Requires threading lastTxnConn/lastTxnLsid through continueTransaction. If the abandoned txn was already aborted server-side (SERVER-128348's failure mode), the commit will fail with NoSuchTransaction; the flusher should tolerate that as a no-op.

      An alternative cleaner boundary: have mochalite.js invoke a TestData.txnOverrideFlushHook if present, and the override registers into it. Touches two files instead of one but avoids the Object.defineProperty intercept.

      Scope

      • Fix targets master; this ticket is not intended as a v8.2 backport candidate.
      • v8.2-staging BF-44810 / BF-44833 will be mitigated separately by adding an after(...) hook to geo_s2stale_polygon_parse.js that issues a non-CRUD command (e.g. {{db.adminCommand( {ping:1}

        )}}), which triggers the override's existing commit path.

      • Should also close out the class of failures represented by BF-43660 once landed.

      Not addressed here

      The interaction with fixture-level parameters (writePeriodicNoops=false, transactionLifetimeLimitSeconds=86400, disableLogicalSessionCacheRefresh=true) that turns an abandoned txn from "annoying" into "hangs for the full task timeout" is orthogonal — worth a separate look, but with this hook in place the abandoned-txn state should not exist in the first place.

            Assignee:
            Charlie Swanson
            Reporter:
            Charlie Swanson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: