ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Concurrency
    • None
    • Storage Engines - Transactions
    • 332.361
    • None
    • None

      __wt_readlock calculates the number of active writers without wrapping it modulo 256. When a writer has ticket 255, the result is negative, and a reader that arrives at this moment sleeps on a condition variable for up to 10 ms instead of queuing behind the writer. This happens once every 256 write locks, while the writer with ticket 255 waits for or holds the lock. The fix computes the difference modulo 256.

      Problem (develop 9d36b5fde6, src/support/mtx_rw.c)

      • Writer tickets current and next are uint8_t (src/include/mutex.h:56-57). A writer takes the ticket next, increments next modulo 256, and holds the lock while current equals its ticket.
      • __wt_readlock (L221) computes writers_active = old.u.s.next - old.u.s.current;. writers_active is int16_t (L183), and the subtraction runs as signed int.
      • While the writer with ticket 255 waits for or holds the lock, current is 255 and next is 0, setting writers_active to -255.
      • readers_queued > writers_active (L222) evaluates to true, sending the reader to stall: __wt_cond_wait on cond_readers with a 10 ms timeout (L224).
      • The reader is not counted in readers_queued, so __wt_writeunlock does not signal cond_readers for it (L523). The reader sleeps up to 10 ms.
      • The sleep occurs before lock wait time is measured (L242), so rwlock statistics do not show it.

      The line comes from WT-3345 (42daa132f2). mongodb-7.0, mongodb-8.0, and mongodb-8.2 have the same code.

      Effect in MongoDB

      All WT_RWLOCK locks have this bug. The locks acquired for write most frequently are the global transaction lock (set_timestamp, __wt_txn_update_oldest) and the dhandle list lock. Their blocking readers include the snapshot at transaction start and the dhandle lookup after a session cache miss. On our MongoDB 8.0 nodes, these locks are taken for write 107-209 times per second on average over 7 days (the counter also includes failed __wt_try_writelock calls), so their ticket wraps every 1.2-2.4 s. A reader sleeps only if it arrives during the window of one write lock out of 256, so stalls are rare. Each stall adds up to 10 ms to one operation.

      Reproduction

      Catch2 test test/catch2/misc_tests/test_rwlock_ticket_wrap.cpp (in the PR): acquires and releases the write lock 255 times, acquires it again with ticket 255, starts a reader thread, and waits up to 10 s for the reader to queue.

      Build Result Runs
      develop 9d36b5fde6 fails: the reader does not queue in 10 s 3 of 3 fail on Linux and on Mac
      develop + change passes in about 2 ms 100 of 100 pass on Linux and on Mac

      Debug builds with HAVE_DIAGNOSTIC=1. Linux (EPYC Genoa, 4 vCPU) and Mac (M4 Pro).

      Measurements

      Standalone benchmark on one WT_RWLOCK (attached). One writer takes the write lock, holds it for the specified time, releases it, and sleeps for the same duration. 8 readers acquire and release the read lock in a loop with a 20 µs pause. 10 s per run, 5 runs. The table shows reads of 5 ms or longer that started while the writer with ticket 255 waited for or held the lock: median (min-max).

      Host Writer hold and sleep develop develop + change
      Linux (EPYC Genoa, 4 vCPU) 200 µs 139 (135-159) 2 (1-5)
      Linux (EPYC Genoa, 4 vCPU) 2000 µs 39 (38-40) 6 (4-8)
      Mac (M4 Pro) 200 µs 592 (398-648) 0 (0-6)
      Mac (M4 Pro) 2000 µs 64 (56-64) 0 (0-0)

      __wt_readlock, __wt_writelock, and __wt_writeunlock in the benchmark match develop 9d36b5fde6.

      Proposed change

      In __wt_readlock, make writers_active a uint8_t and compute (uint8_t)(old.u.s.next - old.u.s.current). __wt_writelock issues at most 255 tickets at once (L415), so the difference modulo 256 is the number of writers that have a ticket.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Mikhail Sotnikov (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: