transaction_aborted_under_cache_pressure.js leaks sessions at teardown and the cache-pressure detector rarely fires under WT-18120's improved eviction

XMLWordPrintableJSON

    • Storage Engines - Transactions
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      jstests/noPassthrough/txns_cache_errors/transaction_aborted_under_cache_pressure.js has been flaky/failing on amazon-linux2023-arm64-static-compile and commit-queue since WT-18120 landed (tracked as BF-45315). Root cause is two long-standing test defects that WT-18120 pushed from rare to frequent — not a WiredTiger bug.

      What fails in CI

      On the mainline failure the cache-pressure mechanism worked correctly (successfulKills > 0), the fill loop exited normally, and the TemporarilyUnavailable assertion passed. The test then died in teardown:

      • The fill loop opens one session per iteration and never calls endSession().
      • After stopSet() kills the server, the shell's GC finalizer retries endSessions against a dead server once per leaked session per second (920 lines of log id 22791 on the failing run).
      • resmoke's hang analyzer SIGABRTs the shell. Task wall clock 1h53m vs ~38s normally.

      Why lowering cache_size does not help

      isUnderCachePressure() (wiredtiger_cache_pressure_monitor.cpp:69) is threadPressureResult && cacheRatioResult. cache_size only affects the cacheRatioResult half; that half was never the blocker — measured at 8M it was satisfied 12x more often than in a successful 16M run, yet the detector never fired in 12,615 fill-loop iterations (vs firing after 1,689 at 16M).

      The binding condition is threadPressureResult, which requires an application thread to be blocked on eviction for ≥95% of every sampling interval (cachePressureEvictionStallThresholdProportion, default 0.95). That is a time condition; cache size cannot produce it.

      Where WT-18120 comes in

      Instrumented before/after runs (same test, same 16M config, only the WT-18120 import commit differs):

      metric before WT-18120 after WT-18120
      iterations to trigger (N) 6 14,289
      test duration 12.0 s 357.9 s
      app-thread eviction stall ≈0.73 s/s ≈0.44 s/s
      detector trigger rate (passes/s) 0.334 0.0056 (59x lower)
      observed/threshold ratio max 0.45 p50 0.49, max 0.88 (bar is 0.95)

      WT-18120 is a genuine eviction improvement — stall time per second dropped. That is exactly why the test breaks: its trigger condition is "an application thread is stalled," and the improvement removes the stall.

      Test-side fix (being landed via the linked PR)

      • sessions[i].endSession() in the drain loop — removes the teardown storm.
      • cachePressureEvictionStallThresholdProportion 0.95→0.3 — measured: 100% of samples sit above 0.4, so 0.95 is unreachable for this single-threaded workload.
      • cachePressureQueryPeriodMilliseconds 1000→20 and CachePressureAbortSessionKillLimitPerBatch 20→1 — the drain loop only lasts a few hundred ms, so at the default period/batch the background killer thread either never completes a cycle or clears the whole backlog before the client's own abort is in flight, so the client never observes TemporarilyUnavailable. Verified: client now observes TemporarilyUnavailable: ... caused by :: Transaction aborted due to cache pressure on 2 of 4 sessions with zero NoSuchTransaction, up from 2 of 1,689 (0.12%) on the mainline run — the assertion passes on its own merits, not by luck. cache_size stays at 16M and the assertion is unchanged (deliberately not relaxed to accept NoSuchTransaction, since that would let a run where cache pressure never fired at all pass silently).

            Assignee:
            Shoufu Du
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: