[SERVER-15902] use signal handling stack via sigaltstack to improve behavior on stack overflow Created: 31/Oct/14  Updated: 09/Oct/20  Resolved: 25/Oct/19

Status: Closed
Project: Core Server
Component/s: Diagnostics, Internal Code
Affects Version/s: None
Fix Version/s: 4.3.1, 4.2.6, 4.0.21

Type: Improvement Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Billy Donahue
Resolution: Done Votes: 1
Labels: platforms-re-triaged, service_architecture_continuous_delivery
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File stacktrace-writeup.txt    
Issue Links:
Backports
Depends
Related
related to SERVER-30738 Implement stdx::thread with control o... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.2, v4.0
Sprint: Dev Tools 2019-08-26, Dev Tools 2019-09-09, Dev Tools 2019-09-23, Dev Tools 2019-10-07, Dev Tools 2019-10-21, Dev Tools 2019-11-04
Participants:

 Description   

In case of stack overflow mongod may not be able to produce a stack trace in the log because the signal handler has no stack space to run in, significantly hampering diagnosis. Servicability could be improved by using sigaltstack to establish a separate signal-handling stack, allowing stack traces to be produced in case of stack overflow.



 Comments   
Comment by Githook User [ 09/Oct/20 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-15902 sigaltstack for workers and stdx::thread.

(cherry picked from commit 0d84ec739b3e831de70775a5cae20ac1c26c28b1)

Comment by Githook User [ 07/Apr/20 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-15902 sigaltstack for workers and stdx::thread.

(cherry picked from commit 0d84ec739b3e831de70775a5cae20ac1c26c28b1)
Branch: v4.2
https://github.com/mongodb/mongo/commit/c0df7ba373a8ef50c9d3a642b9dbaaf3ae2fd7f5

Comment by Githook User [ 25/Oct/19 ]

Author:

{'username': 'BillyDonahue', 'email': 'billy.donahue@mongodb.com', 'name': 'Billy Donahue'}

Message: SERVER-15902 sigaltstack for workers and stdx::thread.
Branch: master
https://github.com/mongodb/mongo/commit/0d84ec739b3e831de70775a5cae20ac1c26c28b1

Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ]

Thanks for pointing that out. Agree that it still seems like a net win - it seems unlikely that a process that close to the memory limit would remain in that state for long before crashing anyway. And it's not clear that the alternative (gdb) would function that well in such a circumstance either.

Comment by Andrew Morrow (Inactive) [ 23/Oct/19 ]

Thanks bruce.lucas. One other interaction of note. I expect that Billy will soon start working on SERVER-33445, which will make it possible to send a signal and have all threads dump their stacks. Under this scenario, when using sigaltstack this would cause some of the altstack memory to be actually allocated on every thread. For a server that was near its memory limits, that could be fatal, as it would force those pages to be backed by real memory, at least briefly. Given that the current alternative is to connect with GDB, I think that is still a net win, but I wanted to point it out.

Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ]

Thanks for the heads-up. I don't see any problem with the Cons, and the Pros look good to me.

Comment by Billy Donahue [ 23/Oct/19 ]

bruce.lucas, we've implemented sigaltstack where available, for stdx::thread and for serviceWorkerThread. We wanted to run it by you for review (https://mongodbcr.appspot.com/505290001).

It's been empirically determined that 16kiB should be enough for our stack trace printer, and we add a 4X safety margin for 64kiB. This adds 6.25% to the virtual memory to the 1MiB main stack allocated for each service worker thread. Due to Linux CoW demand paging, however, this memory won't really be consumed unless the thread uses it by catching a signal, so this added memory is not so bad.

All synchronous signal handlers are going to use these alternative signal stacks now. This will help diagnostic reporting of threads that are running close to the edge of their stack allocations, and of course with the SIGSEGV caused by stack overflows (including deep/ recursion).

Some tradeoffs, as far as I can tell:

Pros:

    - We will be able to begin diagnosing stack overruns, but also diagnose other synchronous signals more reliably if they're running near top of stack or on a corrupted stack.

    - This was a low-impact change to the thread creation and signal handler installation routines.

Cons:

    - Signal handlers will be constrained to 64kiB of stack, rather than the large 1MiB main thread stack. We will install a test to give some confidence that handlers stay under this limit going forward. We might even be able to reduce the 64kiB to 32kiB after some smoke testing.

    - Per-thread virtual memory cost increases by 6.25% (64kiB/1MiB), but this overhead is not paged in until a synchronous signal comes to that thread.

    - sigaltstack doesn't exist on Windows.

    - unwinding stack from a signal handler is potentially more complex when the signal is on a sigaltstack. For example, `backtrace()` on MacOS cannot unwind from a sigaltstack, so we cannot use sigaltstack there.

Comment by Billy Donahue [ 13/Oct/19 ]

stacktrace-writeup.txt

Comment by Andrew Morrow (Inactive) [ 24/Jul/19 ]

Moving this to Q3 quick wins. It doesn't really have anything directly to do with libunwind, and is probably best handled by way of SERVER-30738.

Comment by Andy Schwerin [ 05/Nov/14 ]

This is a modest amount of work, because thread creation occurs in several places in the code base. Catching the most likely offenders (threads handling user connections) would be less work, certainly. bruce.lucas@10gen.com, I'm having trouble bucketing it for planning, so I'm going to leave it in Needs Triage for the time being.

Generated at Thu Feb 08 03:39:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.