[SERVER-35476] Invalid access at address Created: 07/Jun/18  Updated: 16/Jul/18  Resolved: 16/Jul/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dushyant Bangal Assignee: Xiangyu Yao (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 16.04
MongoDB 3.2.20


Attachments: File dushyant_mongod.log    
Operating System: ALL
Sprint: Storage NYC 2018-07-02, Storage NYC 2018-07-16, Storage NYC 2018-07-30
Participants:

 Description   

My mongodb local instance just crashed on its own with the following in its log:

> Invalid access at address: 0x18
>
> Got signal: 11 (Segmentation fault).

*Environment:*

  • Ubuntu 16.04
  • MongoDB v3.2.20

*Backtrace:*

 

----- BEGIN BACKTRACE -----
 {"backtrace":[
{"b":"400000","o":"11599A2","s":"_ZN5mongo15printStackTraceERSo"}
,{"b":"400000","o":"1158949"},{"b":"400000","o":"1159327"},{"b":"7FBC9AD60000","o":"11390"},{"b":"400000","o":"991E41","s":"ZNK5mongo17IndexAccessMethod17getSpaceUsedBytesEPNS_16OperationContextE"},{"b":"400000","o":"73076E","s":"_ZN5mongo10Collection12getIndexSizeEPNS_16OperationContextEPNS_14BSONObjBuilderEi"},{"b":"400000","o":"74BE4E","s":"_ZN5mongo8Database8getStatsEPNS_16OperationContextEPNS_14BSONObjBuilderEd"},{"b":"400000","o":"87EF60","s":"_ZN5mongo7DBStats3runEPNS_16OperationContextERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},{"b":"400000","o":"87A686","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"400000","o":"87B97B","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"400000","o":"786A4B","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"400000","o":"9B4C8A"},{"b":"400000","o":"9B8196","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"5C6F60"},{"b":"400000","o":"10FF8D1","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7FBC9AD60000","o":"76BA"},{"b":"7FBC9A996000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.20", "gitVersion" : "a7a144f40b70bfe290906eb33ff2714933544af8", "compiledModules" : [], "uname" :
{ "sysname" : "Linux", "release" : "4.13.0-43-generic", "version" : "#48~16.04.1-Ubuntu SMP Thu May 17 12:56:46 UTC 2018", "machine" : "x86_64" }
, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "907E25B81823263E277D6DC159426B098C060119" }, { "b" : "7FFEE8DE6000", "elfType" : 3, "buildId" : "E40F36491D2AC74EE1D4F6CBBF5D52F8D60C760C" }, { "b" : "7FBC9BCEC000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "B6FD9C4DDE7FD8CF2DEE25FC19AA41607134E588" }, { "b" : "7FBC9B8A8000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "0081280CE06E36667B03C21564FECD06528E71AD" }, { "b" : "7FBC9B6A0000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7FBC9B49C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7FBC9B193000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7FBC9AF7D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FBC9AD60000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7FBC9A996000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7FBC9BF55000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x15599a2]
 mongod(+0x1158949) [0x1558949]
 mongod(+0x1159327) [0x1559327]
 libpthread.so.0(+0x11390) [0x7fbc9ad71390]
 mongod(_ZNK5mongo17IndexAccessMethod17getSpaceUsedBytesEPNS_16OperationContextE+0x1) [0xd91e41]
 mongod(_ZN5mongo10Collection12getIndexSizeEPNS_16OperationContextEPNS_14BSONObjBuilderEi+0x8E) [0xb3076e]
 mongod(_ZN5mongo8Database8getStatsEPNS_16OperationContextEPNS_14BSONObjBuilderEd+0x39E) [0xb4be4e]
 mongod(ZN5mongo7DBStats3runEPNS_16OperationContextERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x310) [0xc7ef60]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x676) [0xc7a686]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x85B) [0xc7b97b]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x25B) [0xb86a4b]
 mongod(+0x9B4C8A) [0xdb4c8a]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7D6) [0xdb8196]
 mongod(+0x5C6F60) [0x9c6f60]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x311) [0x14ff8d1]
 libpthread.so.0(+0x76BA) [0x7fbc9ad676ba]
 libc.so.6(clone+0x6D) [0x7fbc9aa9d41d]
 ----- END BACKTRACE -----

 

Can someone help me figure out whats the reason behind it?



 Comments   
Comment by Kelsey Schubert [ 20/Jun/18 ]

Hi dushyant,

Would you be able to provide your data files for analysis to help speed our investigation? I'd be happy to to provide a secure upload portal that only MongoDB employees investigating this issue can access. In either case, could you give us a rough estimate of the total size as well as number of databases and collections on the node? As this will help us determine the next diagnostic steps as continue to investigate.

I also have a few more questions about the crash specifically:

  • How frequently does this issue occur? Are you aware of any triggers?
  • Did you observe any unclean shutdowns or other noteworthy events occurred before you began observing the issue?

Lastly, to better understand the issue leading to this segfault, would you please increase loglevel of the query component to 2?

On a running system, you can run db.setLogLevel(2, "query") or, to persist on restart, please enter the following in your config:

systemLog:
   component:
      query:
         verbosity: 2

After the next segfault, would you please upload the complete log file with the more verbose logging?

Thank you for your help,
Kelsey

Comment by Dushyant Bangal [ 14/Jun/18 ]

Hi Nick,

I've attached the log file. I'm not sure if its of the same time I've mentioned in the issue description, but I got it multiple times in the log file.

My mongo has been crashed multiple times now.
I even tried `db.repairDatabase` but I'm still getting the issue.

Comment by Nick Brewer [ 08/Jun/18 ]

Hi Dushyant, 

Thanks for your report. I'd like to see the content of the mongod.log file from the last restart up until the crash - you can attach it to this ticket.

Regards, 

Nick

Generated at Thu Feb 08 04:39:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.