It seems when the node persists new config in collection "local.system.replset", it takes "local" database lock in stronger mode (X) and PBWM lock in IS mode. This can lead to 2 major side effects.
1) Since PBWM lock is taken in IS mode, this can block the secondary oplog applier which requires PBWM in X mode. This can result in replication lag.
2) Since it takes "local" database lock in X mode, this can block other local database readers and writers. This will be addressed by
- Mainly, if this node X is the sync source for node Y, then the oplog fetcher of the node Y can be blocked behind the the reconfig via heartbeat thread due to database lock conflict, leading to replication lag.
Generally, Upserts on non-replicated collection doesn't need to acquire pbwm lock. Basically, storeLocalConfigDocument should acquire DB lock under ShouldNotConflictWithSecondaryBatchApplicationBlock.