[SERVER-59448] Segmentation fault exit Created: 19/Aug/21 Updated: 11/Feb/22 Resolved: 11/Feb/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ivan Takarlikov | Assignee: | Edwin Zhou |
| Resolution: | Done | Votes: | 0 |
| Labels: | Bug | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
We are experiencing repeated Segmentation Fault errors after upgrading to version 4.4.8 from 4.4.7. Here are the lines which are logged at the time of failure.
|
| Comments |
| Comment by Edwin Zhou [ 11/Feb/22 ] | |||||
|
Hi ivan.takarlikov@sensortower.com, Thanks for following up that the collection successfully validating did not prevent the segmentation faults. We're happy to hear that you were able to solve this issue by initial syncing a new node. Since this issue has gone away, I will now close this ticket. Best, | |||||
| Comment by Ivan Takarlikov [ 01/Feb/22 ] | |||||
|
@Eric Sedor, sorry for the delay, yeah, we did checks, a node that was successfully validated, still crashed, but after that, we tried to completely resync it from scratch (I mean to create a new primary on mongo 4.4.10, resync it from scratch from existing primary and remove old primary). And now it works well, no issues from November! So, probably some disk issues were a root cause, we didn't investigate deeper actually. | |||||
| Comment by Eric Sedor [ 31/Jan/22 ] | |||||
|
Hi ivan.takarlikov@sensortower.com, I wanted to follow up to see if you had a chance to validate the collections on the node that crashed in this way. | |||||
| Comment by Ivan Takarlikov [ 18/Oct/21 ] | |||||
|
@Eric Sedor Hey! Thanks for the update. Will check and reach you back! | |||||
| Comment by Eric Sedor [ 18/Oct/21 ] | |||||
|
Hello and thanks for your patience. I apologize for the delay. First, sanjeethgwd@gmail.com that looks like a different issue; if you are still seeing that problem, can you please open a new ticket for it? ivan.takarlikov@sensortower.com, an invalid access message with this backtrace would lead me to suspect corruption within document data. I'm not able to provide guidance on what collection because mongod_pretty.log appears to have been redacted using post-processing (Slow query lines have an empty attr field), but my general guidance would be: 1) Upgrade to 4.4.9 (some corruption issues have been fixed in that version, but I don't believe we should suspect them here as they aren't expected to cause the kind of corruption this issue suggests) Can you let me know if you are able to identify any problematic documents using validate(), or if you can confirm the node that crashes is passing validate() for all collections? | |||||
| Comment by Ivan Takarlikov [ 13/Sep/21 ] | |||||
|
Hey @Eric Sedor ! Any updates on that issue? Did it resolve on 4.4.9? | |||||
| Comment by Sanjeeth Mallesh [ 02/Sep/21 ] | |||||
|
We're encountering same error on `4.2.12` & `4.2.15` version Mongod. Signal 11 is raised and mongo service entering failed/crashed state. Segementation Fault is observed on all other related nodes having same mongod `4.2.12` & `4.2.15`
| |||||
| Comment by Ivan Takarlikov [ 23/Aug/21 ] | |||||
|
Hey @Eric Sedor, uploaded files to uploader as you mentioned. It was Replica set primary node, it happened 4 times within 1 day. The timeline is the following:
Thank you! | |||||
| Comment by Eric Sedor [ 19/Aug/21 ] | |||||
|
Hi ivan.takarlikov@sensortower.com, Would you please archive (tar or zip) the mongod.log files leading up to the first crash and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. As well, can you:
Thank you, |