[SERVER-36514] Hold lock on oplog as soon as optime is reserved Created: 07/Aug/18 Updated: 15/Sep/18 Resolved: 24/Aug/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Spencer Brody (Inactive) | Assignee: | Spencer Brody (Inactive) |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
There exists a potential 3-way deadlock currently. 1. First, perform an insert into a replicated collection using insertDocuments(). An optime is generated, but not committed. If another write transaction commits after this with a later optime, a "hole" is created by the timestamped write that is not yet committed. 2. Someone tries to read the oplog, for example a secondary querying against its sync source. Oplog reads call waitForAllEarlierOplogWritesToBeVisible() to wait for all uncommitted operations to become committed and visible.
3. A DDL operation is received on the "local" database, and enqueues a request for an X lock on the 'local' database. Now the local database DDL operation in thread 3 blocks behind the IX lock held by thread 2. Thread 2 can't complete until the insert in thread 1 completes. And the insert in thread 1 winds up blocking when it goes to acquire an IX lock on the local database in order to write its oplog entry. That lock acquisition is blocked by the pending X lock request by thread 3. This deadlock was made far more likely to hit as part of One way to fix this would be if thread 1 took the lock on the oplog before creating the oplog hole and held on to that lock until the oplog entry was written and the hole removed. |
| Comments |
| Comment by Spencer Brody (Inactive) [ 24/Aug/18 ] |
|
We fixed the deadlock by implementing |
| Comment by Gregory McKeon (Inactive) [ 24/Aug/18 ] |
|
spencer is this still DwS? |
| Comment by Eric Milkie [ 08/Aug/18 ] |
|
Alternatively, we could go the other way and remove the oplog locking in logOp() at the Database and Collection level. |
| Comment by Spencer Brody (Inactive) [ 07/Aug/18 ] |
|
|