[SERVER-36514] Hold lock on oplog as soon as optime is reserved Created: 07/Aug/18  Updated: 15/Sep/18  Resolved: 24/Aug/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Spencer Brody (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-35367 Hold locks in fewer callers of waitFo... Closed
is related to SERVER-36534 Don't acquire locks on oplog when wri... Closed
Participants:

 Description   

There exists a potential 3-way deadlock currently.

1. First, perform an insert into a replicated collection using insertDocuments(). An optime is generated, but not committed. If another write transaction commits after this with a later optime, a "hole" is created by the timestamped write that is not yet committed.

2. Someone tries to read the oplog, for example a secondary querying against its sync source. Oplog reads call waitForAllEarlierOplogWritesToBeVisible() to wait for all uncommitted operations to become committed and visible.

  • This waits for the uncommitted insert in step 1 to be committed while holding an IS lock on the oplog (and thus on the 'local' database).

3. A DDL operation is received on the "local" database, and enqueues a request for an X lock on the 'local' database.

Now the local database DDL operation in thread 3 blocks behind the IX lock held by thread 2. Thread 2 can't complete until the insert in thread 1 completes. And the insert in thread 1 winds up blocking when it goes to acquire an IX lock on the local database in order to write its oplog entry. That lock acquisition is blocked by the pending X lock request by thread 3.

This deadlock was made far more likely to hit as part of SERVER-35365, which changed all mapReduce commands to perform DDL operations on the local database.

One way to fix this would be if thread 1 took the lock on the oplog before creating the oplog hole and held on to that lock until the oplog entry was written and the hole removed.



 Comments   
Comment by Spencer Brody (Inactive) [ 24/Aug/18 ]

We fixed the deadlock by implementing SERVER-36534 instead of this.

Comment by Gregory McKeon (Inactive) [ 24/Aug/18 ]

spencer is this still DwS?

Comment by Eric Milkie [ 08/Aug/18 ]

Alternatively, we could go the other way and remove the oplog locking in logOp() at the Database and Collection level.

Comment by Spencer Brody (Inactive) [ 07/Aug/18 ]

SERVER-35367 describes an alternate approach to fixing the same deadlock.

Generated at Thu Feb 08 04:43:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.