ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • WT12.0.0, 9.1.0-rc0
    • Affects Version/s: None
    • Component/s: Cursors
    • None
    • Storage Engines - Foundations
    • 614.809
    • None
    • None

      Every cursor cache and reopen cycle across all threads currently updates a single shared field (conn->open_cursor_count) using atomic read-modify-writes. This change replaces that shared counter with a per-session field updated without atomics, aggregating it across sessions only when connection statistics are read. This eliminates a cross-thread cache-line bounce during cursor open and close operations while keeping the reported statistic value intact.

      The connection statistic cursor: open cursor count (cursor_open_count) is stored in conn->open_cursor_count (src/include/connection.h:1049, develop at 50c032ba6). Four locations in src/cursor/cur_std.c update it with a relaxed atomic read-modify-write:

      Place Function When Operation
      cur_std.c:760 __wti_cursor_cache a cursor is closed into the session cursor cache __wt_atomic_sub_uint32_relaxed
      cur_std.c:790 __wti_cursor_reopen a cursor is reopened from the session cursor cache __wt_atomic_add_uint32_relaxed
      cur_std.c:1151 __wt_cursor_close a cursor on the session's open list is closed __wt_atomic_sub_uint32_relaxed
      cur_std.c:1550 __wt_cursor_init a cursor is created __wt_atomic_add_uint32_relaxed

      The only reader is __wt_conn_stat_init (src/conn/conn_stat.c:87-88), which runs when a statistics: cursor opens (src/cursor/cur_stat.c:374), including during statistics log server dumps (__statlog_dump, src/conn/conn_stat.c:317). No test reads the statistic since WT-18455.

      Every cursor cache cycle on every thread and table writes to this shared cache line.

      History

      WT-2029 (87bb02959) split connection statistics into per-session buckets updated without atomics. Values requiring exact tracking moved to connection fields: "There are only 3 statistics we maintain atomically: the count of open cursors and the split-stash bytes and object counts. Move them to WT_CONNECTION_IMPL fields, and copy them into the statistics on demand". WT-15754 (685336070) made bucketed connection and data-source statistic updates relaxed atomic read-modify-writes, preventing lost updates. A bucketed statistic remains unsuitable here: it requires a shared atomic read-modify-write on a bucket shared by all sessions with the same id % 23, and updates are skipped when statistics=(none), creating an offset if enabled later via WT_CONNECTION::reconfigure. A counter in the session avoids both issues.

      Why it matters for MongoDB

      With default wiredTigerCursorCacheSize (100), mongod closes cursors when returning a session to the pool and reopens them from the session cursor cache on the next operation. Each operation executes one cache/reopen cycle per touched table (record store and indexes), producing two atomic read-modify-writes on conn>open_cursor_count per table per operation across all threads. SERVER-122455 estimates the close, cache, and reopen cycle at about 1.6% CPU on YCSB in-cache 100% read with 128 threads. mongod exposes this counter in serverStatus().wiredTiger.cursor["open cursor count"] and in FTDC every second for tracking cursor leaks (SERVER-118759), requiring its behavior to remain consistent.

      Measurements

      Microbenchmark configuration: T threads, each with its own session, loop open_cursor (served from the session cursor cache) and close, or open_cursor, search, and close inside one snapshot transaction per thread. Threads access separate tables (distinct) or a single table (shared); in_memory=true, statistics=(fast), 10,000 rows per table, 1 s warm-up, and 2 s measured. Base commit is develop 50c032ba6; patch includes this change; both built with Release (-O2). 5 alternating runs per data point. Values show total M ops/s across all threads, median [min..max]; p is an exact two-sided permutation test on the difference of means over the 5 + 5 runs (0.008 indicates complete separation between builds). Except for one run measuring 1.001, all runs reported 1.0 cursor reopens per operation.

      Operation Tables Threads develop patch Change p
      open, close distinct 1 18.521 [16.095..18.779] 18.258 [17.277..18.579] -1.4% 0.968
      open, close distinct 2 11.118 [10.507..11.237] 35.629 [33.850..37.238] +220.5% 0.008
      open, close distinct 4 10.030 [9.558..10.960] 63.293 [62.597..65.058] +531.0% 0.008
      open, close distinct 8 8.925 [7.875..9.671] 108.872 [96.600..117.378] +1119.8% 0.008
      open, close distinct 12 7.751 [7.553..8.027] 124.087 [98.377..139.268] +1501.0% 0.008
      open, close shared 1 18.676 [17.870..18.988] 18.147 [17.494..18.273] -2.8% 0.048
      open, close shared 2 6.011 [5.690..6.434] 7.908 [7.420..8.267] +31.5% 0.008
      open, close shared 4 4.573 [4.317..5.414] 4.540 [4.452..5.253] -0.7% 0.873
      open, close shared 8 3.531 [2.983..3.716] 2.888 [2.774..3.335] -18.2% 0.040
      open, close shared 12 2.960 [2.510..3.161] 2.819 [2.432..3.280] -4.7% 0.770
      open, search, close in a transaction distinct 1 4.258 [4.159..4.327] 4.150 [4.082..4.374] -2.5% 0.262
      open, search, close in a transaction distinct 2 6.107 [5.831..6.334] 7.601 [7.119..7.756] +24.5% 0.008
      open, search, close in a transaction distinct 4 7.871 [7.288..8.221] 13.021 [12.736..13.587] +65.4% 0.008
      open, search, close in a transaction distinct 8 7.555 [7.326..7.704] 20.363 [19.977..23.181] +169.5% 0.008
      open, search, close in a transaction distinct 12 6.885 [6.717..7.117] 25.125 [20.309..30.724] +264.9% 0.008

      With distinct tables, develop degrades from 2 to 12 threads due to cache contention on conn->open_cursor_count. With one shared table, throughput is gated by data handle atomics (session_inuse, references, handle read lock), yielding variable deltas (+31.5% at 2 threads, -18.2% at 8 threads) that require dedicated machine validation.

      Read cost for cursor: open cursor count through a statistics: cursor (open, search, read, close) with one thread, no workers, and idle sessions with one open/close cycle each (5 runs, microseconds, median [min..max]):

      Idle sessions develop patch Change p Statistic value, both builds
      0 9.92 [9.68..10.99] 9.72 [9.58..10.03] -2.0% 0.230 3
      100 9.88 [9.40..10.36] 10.22 [9.76..10.56] +3.4% 0.294 103
      1000 11.22 [10.93..11.43] 13.04 [12.60..13.36] +16.2% 0.008 1003
      4000 14.44 [14.15..15.79] 22.04 [21.40..22.46] +52.6% 0.008 4003

      The session walk adds roughly 1.9 ns per session slot (+7.6 us at 4000 sessions). Idle sessions remain cached. Active sessions updating cursors may cause cache misses; estimated at ~100 ns per miss, this would add roughly 0.4 ms per read with 4000 sessions, or 0.04% of a core at a 1-second FTDC poll rate.

      Test environment: Apple M4 Pro, 14 cores (10 performance, 4 efficiency), macOS 26.5.2, Apple clang 21.0.0. macOS lacks thread pinning, placing some threads on efficiency cores at 12 threads. Contention is inferred from throughput due to lack of perf c2c. The machine ran concurrent workloads during execution: 1-minute load average 7.91-26.78 (median 17.98) for open/close, 16.99-22.32 for transaction runs, and 15.64-21.15 for statistics latency. Cross-core cache transfers on this single die are faster than multi-socket or large Graviton/x86 architectures. This benchmark isolates the cursor cache path; mongod additionally runs transactions, query parsing, and network I/O.

      Proposed change

      • Replace conn->open_cursor_count with WT_SESSION_IMPL::open_cursor_count. The same four sites update this field. Because these call sites already update the session cursor list without locks, updates use relaxed loads and stores without atomic read-modify-writes.
      • __wt_conn_stat_init iterates over active sessions (including internal sessions) via __wt_session_array_walk to sum the field, matching __wt_gen_oldest and hazard pointer checks.
      • Preserves statistic behavior: updated regardless of statistics setting, excludes cached cursors, and includes internal session cursors and metadata cursors. The aggregation is non-atomic and may diverge from an exact point-in-time snapshot by in-flight cursor state transitions during the walk. It cannot produce negative values.
      • Reading connection statistics accesses one field per session slot up to the active array bound.
      • Diagnostic builds assert that session counters do not drop below zero and equal zero upon session close.
      • Added test_stat19 to verify that the statistic tracks cursors opened, cached, reopened, and closed across distinct sessions.

      Pull request: https://github.com/wiredtiger/wiredtiger/pull/14944

            Assignee:
            Ivan Kochin
            Reporter:
            Mikhail Sotnikov (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: