-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Concurrency
-
None
-
Storage Engines - Transactions
-
332.361
-
None
-
None
__wt_readlock calculates the number of active writers without wrapping it modulo 256. When a writer has ticket 255, the result is negative, and a reader that arrives at this moment sleeps on a condition variable for up to 10 ms instead of queuing behind the writer. This happens once every 256 write locks, while the writer with ticket 255 waits for or holds the lock. The fix computes the difference modulo 256.
Problem (develop 9d36b5fde6, src/support/mtx_rw.c)
- Writer tickets current and next are uint8_t (src/include/mutex.h:56-57). A writer takes the ticket next, increments next modulo 256, and holds the lock while current equals its ticket.
- __wt_readlock (L221) computes writers_active = old.u.s.next - old.u.s.current;. writers_active is int16_t (L183), and the subtraction runs as signed int.
- While the writer with ticket 255 waits for or holds the lock, current is 255 and next is 0, setting writers_active to -255.
- readers_queued > writers_active (L222) evaluates to true, sending the reader to stall: __wt_cond_wait on cond_readers with a 10 ms timeout (L224).
- The reader is not counted in readers_queued, so __wt_writeunlock does not signal cond_readers for it (L523). The reader sleeps up to 10 ms.
- The sleep occurs before lock wait time is measured (L242), so rwlock statistics do not show it.
The line comes from WT-3345 (42daa132f2). mongodb-7.0, mongodb-8.0, and mongodb-8.2 have the same code.
Effect in MongoDB
All WT_RWLOCK locks have this bug. The locks acquired for write most frequently are the global transaction lock (set_timestamp, __wt_txn_update_oldest) and the dhandle list lock. Their blocking readers include the snapshot at transaction start and the dhandle lookup after a session cache miss. On our MongoDB 8.0 nodes, these locks are taken for write 107-209 times per second on average over 7 days (the counter also includes failed __wt_try_writelock calls), so their ticket wraps every 1.2-2.4 s. A reader sleeps only if it arrives during the window of one write lock out of 256, so stalls are rare. Each stall adds up to 10 ms to one operation.
Reproduction
Catch2 test test/catch2/misc_tests/test_rwlock_ticket_wrap.cpp (in the PR): acquires and releases the write lock 255 times, acquires it again with ticket 255, starts a reader thread, and waits up to 10 s for the reader to queue.
| Build | Result | Runs |
|---|---|---|
| develop 9d36b5fde6 | fails: the reader does not queue in 10 s | 3 of 3 fail on Linux and on Mac |
| develop + change | passes in about 2 ms | 100 of 100 pass on Linux and on Mac |
Debug builds with HAVE_DIAGNOSTIC=1. Linux (EPYC Genoa, 4 vCPU) and Mac (M4 Pro).
Measurements
Standalone benchmark on one WT_RWLOCK (attached). One writer takes the write lock, holds it for the specified time, releases it, and sleeps for the same duration. 8 readers acquire and release the read lock in a loop with a 20 µs pause. 10 s per run, 5 runs. The table shows reads of 5 ms or longer that started while the writer with ticket 255 waited for or held the lock: median (min-max).
| Host | Writer hold and sleep | develop | develop + change |
|---|---|---|---|
| Linux (EPYC Genoa, 4 vCPU) | 200 µs | 139 (135-159) | 2 (1-5) |
| Linux (EPYC Genoa, 4 vCPU) | 2000 µs | 39 (38-40) | 6 (4-8) |
| Mac (M4 Pro) | 200 µs | 592 (398-648) | 0 (0-6) |
| Mac (M4 Pro) | 2000 µs | 64 (56-64) | 0 (0-0) |
__wt_readlock, __wt_writelock, and __wt_writeunlock in the benchmark match develop 9d36b5fde6.
Proposed change
In __wt_readlock, make writers_active a uint8_t and compute (uint8_t)(old.u.s.next - old.u.s.current). __wt_writelock issues at most 255 tickets at once (L415), so the difference modulo 256 is the number of writers that have a ticket.