-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Items 2 and 3 of SERVER-138793. ServerParameter::set() runs validate, store and on_update with no lock across them (server_parameter_with_storage.h:291-303), and CmdSet takes no lock around it. The two load-shedding knobs rely on that sequence being atomic, and it is not.
Start/stop lost update. onUpdateQueryMemoryLoadSheddingEnablement acts on the newPercent of its own call, and startQueryMemoryRssMonitor checks sheddingActive() before taking holder.mutex. If a disable stores -1, a later enable stores 50 and finishes its hook, and the disable's hook then runs, stop runs after start. Result: the config says enabled, the monitor is stopped, rssBytes is -1 and serverStatus shows active:false. Nothing heals this until the next toggle. The other two outcomes listed in SERVER-138793 (disabled in config with the monitor running, and a fresh RSS sample overwritten by rssBytes.store(-1) after the mutex is released) were not tested.
Cross-validators. validateQueryMemoryLoadSheddingLowMark and validateQueryMemoryLoadSheddingHighMark each check the other knob's current value. Starting from low=80, high=95, one call sets low=90 and another sets high=85. Both validate against the old counterpart and pass, and the stored pair is low=90, high=85. shedProbability returns 0 for high <= low, so shedding is disarmed while serverStatus shows active:true. This contradicts the comment at query_memory_load_shedding.h:115-118.
Reproduced with unit tests that replay the interleaving step by step on one thread (Item2a and Item3 in query_memory_load_shedding_test.cpp, attached in a comment). Both fail on current master. A real-thread stress test (2000 rounds of two threads toggling the knob) did not hit the race, because thread start-up serializes the two calls. The race needs two concurrent setParameter calls on these knobs.
Proposed fix, using one mutex for all config changes of this feature:
- Add a reconcileMonitor() that takes the mutex, derives the wanted state from sheddingActive(), and starts or stops to match. The rssBytes reset moves into the same critical section.
- Add on_update hooks on both marks that take the same mutex and re-check low < high against the stored values. On an invalid pair the hook stores back its knob's last good value and returns BadValue, so one of the two calls fails and the stored pair stays valid.
Open points: between the store and the rollback, shedProbability can briefly see the invalid pair (it returns 0, so the effect is a short gap in shedding). I have not checked whether on_update also fires when startup config is applied, which affects the rejection path. A ServerParameter subclass whose set() holds the mutex across validate and store would avoid the transient state and the rollback, at the cost of more IDL and boilerplate.
- related to
-
SERVER-139122 Query-memory load shedding: fix double count, log outside client locks, trim RSS read under mutex, correct "never throws" comment
-
- In Code Review
-