[SERVER-82974] Segfault using 3.6.17 with mmapv1 Created: 08/Nov/23  Updated: 08/Nov/23  Resolved: 08/Nov/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dan J Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Seeing the following segfault, this is happening quite frequently on primary only. The mapped+journal is around 1T and half for mapped, resident is around 230GB and virtual mem usage is around 1.00T. The server has 256GB physical and is a dedicated mongo machine.  It looks like the segfault is happening when creating NewObject based on the addr2line output below.  

Would appreciate any kind of input to reveal the problem. 

 

 0x56445122738a 0x564451226caf 0x564451226e9c 0x5644502eeb22 0x7f3e6cb6ec30 0x56445085071c 0x56445083cef7 0x5644508554fb 0x564450855a4e 0x5644508565b0 0x5644505f3aa4 0x56445084929> ----- BEGIN BACKTRACE ----- {"backtrace":[\{"b":"56444EF31000","o":"22F638A","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"56444EF31000","o":"22F5CAF"},\{"b":"56444EF31000","o":"22F5E9C"},{"b":"56444EF31000","o">  mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x56445122738a]  mongod(+0x22F5CAF) [0x564451226caf]  mongod(+0x22F5E9C) [0x564451226e9c]  mongod(+0x13BDB22) [0x5644502eeb22]  libpthread.so.0(+0x12C30) [0x7f3e6cb6ec30]  mongod(_ZN2js14TenuringTracer8traverseI8JSObjectEEvPPT_+0xC) [0x56445085071c]  mongod(_ZN2js8frontend9ObjectBox5traceEP8JSTracer+0x57) [0x56445083cef7]  mongod(_ZN2JS12AutoGCRooter8traceAllEP8JSTracer+0xEB) [0x5644508554fb]  mongod(_ZN2js2gc9GCRuntime11markRuntimeEP8JSTracerNS1_18TraceOrMarkRuntimeE+0xAE) [0x564450855a4e]  mongod(_ZN2js7Nursery7collectEP9JSRuntimeN2JS8gcreason6ReasonEPN7mozilla6VectorIPNS_11ObjectGroupELm0ENS_17SystemAllocPolicyEEE+0x290) [0x5644508565b0]  mongod(_ZN2js2gc9GCRuntime7minorGCEP9JSContextN2JS8gcreason6ReasonE+0x84) [0x5644505f3aa4]  mongod(_ZN2js8AllocateI8JSObjectLNS_7AllowGCE1EEEPS1_PNS_16ExclusiveContextENS_2gc9AllocKindEmNS6_11InitialHeapEPKNS_5ClassE+0x11E) [0x56445084929e]  mongod(+0x170E130) [0x56445063f130]  mongod(_ZN2js29NewObjectWithClassProtoCommonEPNS_16ExclusiveContextEPKNS_5ClassEN2JS6HandleIP8JSObjectEENS_2gc9AllocKindENS_13NewObjectKindE+0x204) [0x564450640f24]  mongod(+0x1605AE9) [0x564450536ae9]  mongod(+0x16062B6) [0x5644505372b6]  mongod(+0x16072EE) [0x5644505382ee]  mongod(+0x162652E) [0x56445055752e]  ??? [0x84e8829882d] -----  END BACKTRACE  -----

 

 

mongo1 /var/tmp/mongodb-linux-x86_64-3.6.17 # addr2line -e mongod.debug -ifC 0x162652E
JS::Rooted<JSObject*>::~Rooted()
/data/mci/b2052c4acc8cc5829d7618b735c7995b/src/src/third_party/mozjs-45/include/js/RootingAPI.h:720
NewObject
/data/mci/b2052c4acc8cc5829d7618b735c7995b/src/src/third_party/mozjs-45/extract/js/src/jsobj.cpp:673
  

db.serverBuildInfo()
{
        "version" : "3.6.17",
        "gitVersion" : "3d6953c361213c5bfab23e51ab274ce592edafe6",
        "modules" : [ ],
        "allocator" : "tcmalloc",
        "javascriptEngine" : "mozjs",
        "sysInfo" : "deprecated",
        "versionArray" : [
                3,
                6,
                17,
                0
        ],
        "openssl" : {
                "running" : "OpenSSL 1.1.1i  8 Dec 2020",
                "compiled" : "OpenSSL 1.1.1i  8 Dec 2020"
        },
        "buildEnvironment" : {
                "distmod" : "",
                "distarch" : "x86_64",
                "cc" : "x86_64-pc-linux-gnu-gcc: x86_64-pc-linux-gnu-gcc (Gentoo 9.3.0-r1 p3) 9.3.0",
                "ccflags" : "-fno-omit-frame-pointer -fno-strict-aliasing -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp",
                "cxx" : "x86_64-pc-linux-gnu-g++: x86_64-pc-linux-gnu-g++ (Gentoo 9.3.0-r1 p3) 9.3.0",
                "cxxflags" : "-frecord-gcc-switches -fno-omit-frame-pointer -g -pipe -Woverloaded-virtual -frecord-gcc-switches -fno-omit-frame-pointer -g -pipe -Wpessimizing-move -Wredundant-move -Wno-maybe-uninitialized -std=c++14",
                "linkflags" : "-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -pthread -Wl,-z,now -rdynamic -fstack-protector-strong -fuse-ld=gold -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro",
                "target_arch" : "x86_64",
                "target_os" : "linux"
        },
        "bits" : 64,
        "debug" : false,
        "maxBsonObjectSize" : 16777216,
        "storageEngines" : [
                "devnull",
                "ephemeralForTest",
                "mmapv1",
                "wiredTiger"
        ],
        "ok" : 1
}

Kernel version 5.10.68
libc: 2.32
numPages: 65880603,
maxOpenFiles: 32768



 Comments   
Comment by Chris Kelly [ 08/Nov/23 ]

snowboarddj@gmail.com,

Unfortunately, there's not much we can do to help with your issue on this unsupported version. Specifically:

  • A significant number of changes have been made to MongoDB since 3.6 that likely will resolve your issue. From a quick search, here's just a few examples of segmentation faults on these old versions:
  • Even if we were to discover what caused this issue for you, it would not be fixed on this version.

It's possible that you may come across useful information on other, old SERVER tickets in our Jira. Beyond that, I'd have to recommend you to start by asking our community for help by checking the MongoDB Developer Community Forums.

If the discussion there leads you to suspect a bug in a supported version of MongoDB (4.4+), then we'd want to investigate it as a possible bug here in the SERVER project.

 

Comment by Dan J [ 08/Nov/23 ]

chris.kelly@mongodb.com There are some obstacles to upgrading the Mongo at the moment to V4.4+ for us and is not an option I can explore. Our first step would be to determine if this could be resolved without upgrading and if you can help shed some lights on the cause of the current issue, I would be very much appreciated. 

Comment by Chris Kelly [ 08/Nov/23 ]

Hi snowboarddj@gmail.com,

MongoDB 3.6 reached end of life in April 2021 and is no longer supported. Can you please upgrade to MongoDB v4.4+ and see if your issue persists? Notably, we have long since migrated from MMAP to the WiredTiger storage engine.

Christopher

Generated at Thu Feb 08 06:50:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.