[SERVER-61334] Replication batcher uninterruptible lock deadlocks with storage change Created: 09/Nov/21  Updated: 29/Oct/23  Resolved: 11/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0

Type: Bug Priority: Major - P3
Reporter: Matthew Russotto Assignee: Matthew Russotto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Replication 2021-11-15
Participants:
Linked BF Score: 135

 Description   

The storage change code takes a global lock, interrupts all opCtxs, and waits for the opCtx to be destroyed. An uninterruptible global lock can deadlock with this, since the global lock won't release until the opCtx is destroyed, and the opCtx will wait forever for it. Most uses of uninterruptible locks do not run during initial sync (e.g. prepared transactions) and aren't a problem, but for some reason the ReplBatcher thread runs continuously even in initial sync, and it requires an uninterruptible lock.

Can be fixed by having the batcher take an interruptible global lock before the uninterruptible section.



 Comments   
Comment by Githook User [ 01/Dec/21 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-61792 Extending lock in batcher hurts performance

Before SERVER-61334, we took RSTL IX, PBWM IS, Global IS, once, and released those almost
immediately. SERVER-61334 took RSTL IX, PBWM IS, and Global IS, then took the same locks
recursively, and held them a little longer (not much! It's just getting BSON objects off an
in-memory queue). Global IS is nearly always uncontended (except shutdown and storage change), as is
RSTL IX, and getting a lock we already have is very cheap. But holding PBWM IS a little longer was
enough to essentially serialize oplog application and batching a lot of the time.

This fix takes RSTL IX and Global IS, then takes those locks recursively, and holds them a bit
longer than the original, but there's no contention. The purpose of SERVER-61334 was to ensure we
did not enqueue an uninterruptible global lock while a storage change held Global X, as that results
in deadlock; this purpose is preserved by this change.
Branch: master
https://github.com/mongodb/mongo/commit/526f24e10905eb80f58dcd1ddd5db37853c91a60

Comment by Githook User [ 11/Nov/21 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-61334 Replication batcher uninterruptible lock deadlocks with storage change
Branch: master
https://github.com/mongodb/mongo/commit/f118314d40643af9b2f89fc8dbec0a18e0ddc3c1

Generated at Thu Feb 08 05:52:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.