[SERVER-38162] Acquire ReplicationStateTransitionLock on shutdown in mode X Created: 15/Nov/18 Updated: 29/Oct/23 Resolved: 17/Jan/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.8 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Samyukta Lanka | Assignee: | Samyukta Lanka |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | prepare_durability | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | Repl 2018-12-17, Repl 2019-01-14, Repl 2019-01-28 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Comments |
| Comment by Githook User [ 17/Jan/19 ] |
|
Author: {'username': 'lankas', 'email': 'samy.lanka@mongodb.com', 'name': 'Samy Lanka'}Message: |
| Comment by Siyuan Zhou [ 05/Dec/18 ] |
|
There are places where canAcceptWritesFor is called after holding locks. The contract used to be if this check is true, writing into oplog will succeed. Adding checkForInterrupt() in lopOp breaks the contract. For example, renameCollection and dropCollection follow this pattern. MultiIndexBlockImpl is another example where UninterruptibleLockGuard is used to avoid interruption of writing an oplog entry. I think we need to audit all 47 occurrences of UninterruptibleLockGuard in production code. Not all of them can conflict with transaction locks, e.g. asking for IS or IX modes, working on a temp collection, or writing oplog entries. If we can fix those DDL that require strong locks to be killable, these non-conflicting operations can go in after killing all operations. 18 occurrences are for sharding that I haven't look into yet. One noticeable conflicting case is CollectionCriticalSection. |
| Comment by Samyukta Lanka [ 05/Dec/18 ] |
|
After talking with tess.avitabile and judah.schvimer, we are going to keep the shutdown order the same as what I proposed above. Instead of preventing DDL operations from acquiring locks released by prepared transactions, we will make sure that they can't write an oplog entry if they were killed. To do this, we plan to check for interrupts during logOp, in a place that already uasserts if a write is not allowed (so we know that it is interruptible there). |
| Comment by Samyukta Lanka [ 05/Dec/18 ] |
|
Sorry I forgot to update that, the new proposed order (without Tess's idea) is: |
| Comment by Judah Schvimer [ 05/Dec/18 ] |
|
On a separate note, samy.lanka, as you pointed out, steps 3 and 4 should be swapped, correct? |
| Comment by Tess Avitabile (Inactive) [ 05/Dec/18 ] |
|
Is it possible to wait to abort prepared transactions until after the RSTL is acquired? This means that in step (3), we would need to just abort non-prepared transactions. |
| Comment by Samyukta Lanka [ 04/Dec/18 ] |
|
siyuan.zhou and I were discussing whether we have actually solved the problem of DDL operations sneaking in after aborting prepared transactions on shutdown and we think that they are still a problem even after this work goes in. Some DDL operations are protected by an UninterruptibleLockGuard like dropDatabase, which means that they won't have been killed when we try to kill all operations. This means that after we have aborted all transactions, they could still succeed. |
| Comment by Judah Schvimer [ 03/Dec/18 ] |
|
Per discussion with kaloian.manassiev and siyuan.zhou, before step 3 above we will need to implement a way to release the resources of all sessions, regardless of if they were markedUnkillable. |
| Comment by Samyukta Lanka [ 30/Nov/18 ] |
|
The current order of shutdown is: This is what the new order will be: |
| Comment by Siyuan Zhou [ 29/Nov/18 ] |
|
Just want to add a comment to remind ourselves that it should be impossible to let dropCollections sneak in after aborting prepared transactions on shutdown. |