[SERVER-38791] Invalid access at address: 0 Segmentation fault (signal 11) Created: 28/Dec/18  Updated: 08/Jan/24  Resolved: 05/Feb/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Shahzeb Haidar Assignee: Danny Hatcher (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We are running MongoDB (v 3.6.6) in replication mode (1 primary and 1 secondary) on two nodes (running Ubuntu 14.04.5 LTS,  64GB memory).
We experienced the following error on primary instance:

2018-12-26T19:22:24.491-0500 F -        [conn111] Invalid access at address: 0
2018-12-26T19:22:24.808-0500 F -        [conn111] Got signal: 11 (Segmentation fault).
 0x7ffc4cdb78b1 0x7ffc4cdb6ac9 0x7ffc4cdb7136 0x7ffc495b3330 0x7ffc4b790953 0x7ffc4b7910b5 0x7ffc4b7935e5 0x7ffc4b793edb 0x7ffc4b7922bd 0x7ffc4b795671 0x7ffc4c6a6e05 0x7ffc4cc6b5b4 0x7ffc495ab184 0x7ffc492d803d
----- BEGIN BACKTRACE -----

{"backtrace":[

{"b":"7FFC4AB64000","o":"22538B1","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"7FFC4AB64000","o":"2252AC9"},{"b":"7FFC4AB64000","o":"2253136"},{"b":"7FFC495A3000","o":"10330"},{"b":"7FFC4AB64000","o":"C2C953"},{"b":"7FFC4AB64000","o":"C2D0B5","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"7FFC4AB64000","o":"C2F5E5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"7FFC4AB64000","o":"C2FEDB","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"7FFC4AB64000","o":"C2E2BD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7FFC4AB64000","o":"C31671"},{"b":"7FFC4AB64000","o":"1B42E05"},{"b":"7FFC4AB64000","o":"21075B4"},{"b":"7FFC495A3000","o":"8184"},{"b":"7FFC491DA000","o":"FE03D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.6", "gitVersion" : "6405d65b1d6432e138b44c13085d0c2fe235d6bd", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.13.0-24-generic", "version" : "#47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014", "machine" : "x86_64" }

, "somap" : [ { "b" : "7FFC4AB64000", "elfType" : 3, "buildId" : "F63278FD698B5843222FE7A6C8FF17D6AEFBBE38" }, { "b" : "7FFF52AFE000", "elfType" : 3, "buildId" : "6755FAD2CADACDF1667E5B57FF1EDFC28DD1C976" }, { "b" : "7FFC4A724000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "AD304AFCE6847F7A4D66D22853E87CCBF5A66966" }, { "b" : "7FFC4A348000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "CE5EE930D4F0B1F47EDFDACC388EAC6C4DE5CDD2" }, { "b" : "7FFC4A0E9000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "920BD37B19B7BD04CA38CE35155D6CDCD744EAB5" }, { "b" : "7FFC49EE5000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "034D6A4EE9DCAB4A34ABD644345CBBB42DC63088" }, { "b" : "7FFC49CDD000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "4F930712D3609C93E380E5BE5DF73E7AD273531C" }, { "b" : "7FFC499D7000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "300C7884CDEB5667BEA2357D2B8E7A76397562D6" }, { "b" : "7FFC497C1000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7FFC495A3000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "F64B8AD471FBA1B7A3A64EFB01551E694975E1F7" }, { "b" : "7FFC491DA000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "D9A10B8EF90300628DD0A3A535106967714D7328" }, { "b" : "7FFC4A93F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2CA513EDC89C7BC06EC183D1A3A03CC0F606319C" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7ffc4cdb78b1]
 mongod(+0x2252AC9) [0x7ffc4cdb6ac9]
 mongod(+0x2253136) [0x7ffc4cdb7136]
 libpthread.so.0(+0x10330) [0x7ffc495b3330]
 mongod(+0xC2C953) [0x7ffc4b790953]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x175) [0x7ffc4b7910b5]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x7ffc4b7935e5]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x7ffc4b793edb]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x7ffc4b7922bd]
 mongod(+0xC31671) [0x7ffc4b795671]
 mongod(+0x1B42E05) [0x7ffc4c6a6e05]
 mongod(+0x21075B4) [0x7ffc4cc6b5b4]
 libpthread.so.0(+0x8184) [0x7ffc495ab184]
 libc.so.6(clone+0x6D) [0x7ffc492d803d]
-----  END BACKTRACE  -----
 

NOTE: I am attaching the mongo diagnostic data



 Comments   
Comment by Danny Hatcher (Inactive) [ 05/Feb/19 ]

I'm going to close this as "Cannot Reproduce" for now. Please let us know if the issue reoccurs.

Comment by Danny Hatcher (Inactive) [ 23/Jan/19 ]

Hello Shahzeb,

We appreciate your patience as we've looked into the error but nothing stands out as something we can point to as the root cause. There's nothing further we can do unless the segfault happens again. If it does reoccur, can you please provide the same information you did in this case (the mongod logs and "diagnostic.data")? We would be more than happy to take a look.

Thank you,

Danny

Comment by Shahzeb Haidar [ 23/Jan/19 ]

Hi,

We have not encountered the segfault again. It was a one time occurrence. This ticket was raised to gain insight into any possible cause and to safeguard against any such crash in future.

Thanks,

Shahzeb

Comment by Danny Hatcher (Inactive) [ 22/Jan/19 ]

Hello Shahzeb,

Unfortunately, we’ve been unable to discern a concrete root cause from the information provided. Have you experienced the segfault again or was it a one-time occurrence? If it has reoccured, please provide any context around the new instance including mongod logs covering the incident.

 Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 16/Jan/19 ]

Hello Shahzeb,

We are still looking into this. Thank you for your patience.

Danny

Comment by Shahzeb Haidar [ 15/Jan/19 ]

Hi,
Is there any update on this?

Thanks,

Shahzeb

Comment by Shahzeb Haidar [ 28/Dec/18 ]

Hi Daniel,

I have uploaded the mongod log.

Thanks,

Shahzeb

Comment by Shahzeb Haidar [ 28/Dec/18 ]

Hi,
I will be providing you the logs but it will take some time. Sorry for delay from my side.

Thanks,

Shahzeb

Comment by Danny Hatcher (Inactive) [ 28/Dec/18 ]

Hello Shahzeb,

Can you please also upload the mongod log to that folder? It is located where you have specified your logpath parameter in the mongod configuration.

Thank you,

Danny

Comment by Shahzeb Haidar [ 28/Dec/18 ]

Hi Daniel,

I have uploaded the relevant files from diagnosis.data folder. The name of zip file is mongo_diag_data.zip .
Please tell me if you are looking for some other log files as well.

Thanks,

Shahzeb

Comment by Danny Hatcher (Inactive) [ 28/Dec/18 ]

Hello Shahzeb,

Please provide the full mongod log from the node that crashed so that we can investigate. You can upload the relevant files to our Secure Upload Portal. Please note that only MongoDB Engineers will be able to read the files and they will be automatically deleted after some time.

Thank you,

Danny

Generated at Thu Feb 08 04:50:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.