[SERVER-15902] use signal handling stack via sigaltstack to improve behavior on stack overflow Created: 31/Oct/14 Updated: 09/Oct/20 Resolved: 25/Oct/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Diagnostics, Internal Code |
| Affects Version/s: | None |
| Fix Version/s: | 4.3.1, 4.2.6, 4.0.21 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | Billy Donahue |
| Resolution: | Done | Votes: | 1 |
| Labels: | platforms-re-triaged, service_architecture_continuous_delivery | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v4.2, v4.0
|
||||||||||||||||
| Sprint: | Dev Tools 2019-08-26, Dev Tools 2019-09-09, Dev Tools 2019-09-23, Dev Tools 2019-10-07, Dev Tools 2019-10-21, Dev Tools 2019-11-04 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
In case of stack overflow mongod may not be able to produce a stack trace in the log because the signal handler has no stack space to run in, significantly hampering diagnosis. Servicability could be improved by using sigaltstack to establish a separate signal-handling stack, allowing stack traces to be produced in case of stack overflow. |
| Comments |
| Comment by Githook User [ 09/Oct/20 ] |
|
Author: {'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}Message: (cherry picked from commit 0d84ec739b3e831de70775a5cae20ac1c26c28b1)
|
| Comment by Githook User [ 07/Apr/20 ] |
|
Author: {'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}Message: (cherry picked from commit 0d84ec739b3e831de70775a5cae20ac1c26c28b1) |
| Comment by Githook User [ 25/Oct/19 ] |
|
Author: {'username': 'BillyDonahue', 'email': 'billy.donahue@mongodb.com', 'name': 'Billy Donahue'}Message: |
| Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ] |
|
Thanks for pointing that out. Agree that it still seems like a net win - it seems unlikely that a process that close to the memory limit would remain in that state for long before crashing anyway. And it's not clear that the alternative (gdb) would function that well in such a circumstance either. |
| Comment by Andrew Morrow (Inactive) [ 23/Oct/19 ] |
|
Thanks bruce.lucas. One other interaction of note. I expect that Billy will soon start working on |
| Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ] |
|
Thanks for the heads-up. I don't see any problem with the Cons, and the Pros look good to me. |
| Comment by Billy Donahue [ 23/Oct/19 ] |
|
bruce.lucas, we've implemented sigaltstack where available, for stdx::thread and for serviceWorkerThread. We wanted to run it by you for review (https://mongodbcr.appspot.com/505290001). It's been empirically determined that 16kiB should be enough for our stack trace printer, and we add a 4X safety margin for 64kiB. This adds 6.25% to the virtual memory to the 1MiB main stack allocated for each service worker thread. Due to Linux CoW demand paging, however, this memory won't really be consumed unless the thread uses it by catching a signal, so this added memory is not so bad. All synchronous signal handlers are going to use these alternative signal stacks now. This will help diagnostic reporting of threads that are running close to the edge of their stack allocations, and of course with the SIGSEGV caused by stack overflows (including deep/ recursion). Some tradeoffs, as far as I can tell: Pros: - We will be able to begin diagnosing stack overruns, but also diagnose other synchronous signals more reliably if they're running near top of stack or on a corrupted stack. - This was a low-impact change to the thread creation and signal handler installation routines. Cons: - Signal handlers will be constrained to 64kiB of stack, rather than the large 1MiB main thread stack. We will install a test to give some confidence that handlers stay under this limit going forward. We might even be able to reduce the 64kiB to 32kiB after some smoke testing. - Per-thread virtual memory cost increases by 6.25% (64kiB/1MiB), but this overhead is not paged in until a synchronous signal comes to that thread. - sigaltstack doesn't exist on Windows. - unwinding stack from a signal handler is potentially more complex when the signal is on a sigaltstack. For example, `backtrace()` on MacOS cannot unwind from a sigaltstack, so we cannot use sigaltstack there. |
| Comment by Billy Donahue [ 13/Oct/19 ] |
| Comment by Andrew Morrow (Inactive) [ 24/Jul/19 ] |
|
Moving this to Q3 quick wins. It doesn't really have anything directly to do with libunwind, and is probably best handled by way of |
| Comment by Andy Schwerin [ 05/Nov/14 ] |
|
This is a modest amount of work, because thread creation occurs in several places in the code base. Catching the most likely offenders (threads handling user connections) would be less work, certainly. bruce.lucas@10gen.com, I'm having trouble bucketing it for planning, so I'm going to leave it in Needs Triage for the time being. |