[SERVER-23072] Missing stack trace when mongod crashes with heap corruption Created: 09/Mar/16 Updated: 14/Apr/16 Resolved: 11/Apr/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Diagnostics |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Charlie Swanson | Assignee: | Mark Benvenuto |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Sprint: | Platforms 12 (04/01/16), Platforms 13 (04/22/16) | ||||||||
| Participants: | |||||||||
| Description |
|
In evergreen we had a failure of csrs_upgrade_during_count.js where a mongod crashed with the following error:
The lack of a stack trace makes it difficult to debug what actually went wrong. sharding_WT failed on windows-64-2k8-debugcsrs_upgrade_during_count.js - Logs | History |
| Comments |
| Comment by Mark Benvenuto [ 11/Apr/16 ] | ||||||||||||||||||
|
I have not seen any more reports or a repro. | ||||||||||||||||||
| Comment by Mark Benvenuto [ 18/Mar/16 ] | ||||||||||||||||||
|
Actually, here is the sequence of what happened in this code since this is 3.2: 1. The C RTL detects a problem and calls crtDebugCallback(). I do not know why the stack trace was not printed. It definitely reached this part of the code. I may have stalled doing the I/O to read the symbols from the PDB. The entire log ends < 4 seconds later. In a quick test where injected a similar fault by hand, it worked fine.
| ||||||||||||||||||
| Comment by Eric Milkie [ 10/Mar/16 ] | ||||||||||||||||||
|
0x80000003 is the exception used to stop the debugger, when attached. The debugger installs a top-level exception handler to capture this exception. In the above issue, what I believe should have happened is this: The log above shows that steps 1-4 seem to have worked as expected, but step 5 somehow called breakpoint() first – and IsDebuggerPresent() erroneously returned "true"? Or else something entirely different happened. I'm not sure. | ||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 09/Mar/16 ] | ||||||||||||||||||
|
Could be | ||||||||||||||||||
| Comment by Spencer Brody (Inactive) [ 09/Mar/16 ] | ||||||||||||||||||
|
Hmm, one of the shard mongods crashed:
Unfortunately there doesn't seem to actually be a stack trace anywhere... |