[SERVER-42244] Dump LockManager on shutdown and via a command Created: 15/Jul/19 Updated: 08/Jan/24 Resolved: 09/Oct/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Siyuan Zhou | Assignee: | Mira Carey |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
When a deadlock occurs, it would be helpful to dump the lock manager to help diagnose the deadlock. Ideally, we could also dump the stack trace of the threads in the deadlock. This will be similar to the hang analyzer which uses GDB. |
| Comments |
| Comment by Geert Bosch [ 03/Feb/20 ] |
|
I'm very skeptical about doing anything that requires locking or similar in unclean shutdown. When we hit an invariant, the system is in a state where the best thing we can do is crash quickly. If we need to wait for a mutex, we depend on other threads making progress while we're in the process of crashing. We may creating deadlocks purely as result of this extra locking, or end up with a half hung process rather than a crashed process, which may be worse. |
| Comment by Eric Milkie [ 10/Oct/19 ] |
|
Where's the epic? |
| Comment by Mira Carey [ 09/Oct/19 ] |
|
The main problem here is that you either take a global lock to dump the process before global shutdown (which may prevent successful stepdowns), or you take it after you've killed all the user ops. Where you've probably already broken the deadlock by killing all the ops. We need to do more investigations about how to dump interesting information about locks held, but we'll have to defer that to a future epic about improved diagnosibility of hangs. |
| Comment by Ratika Gandhi [ 19/Aug/19 ] |
|
Siyuan and Jason discussed about creating an epic to track this work. mira.carey@mongodb.com will add the details of the conversation here. |
| Comment by Ratika Gandhi [ 05/Aug/19 ] |
|
Will be discussed a part of the 'Make Mongodb Diagnostics and observability best in class' tactical goal. |