Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16993

mongod crashes with Got signal: 11 (Segmentation fault)

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.8.0-rc5
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL
      • See above for detailed system information and database information; I can provide additional logs or information
      1. queue up workload as described

      I have an ETL workload that I have been testing a 2.8.0 WiredTiger single mongod deployment with. I have been able to reproduce this crash twice. As far as I can tell, there is no data loss after the crash. The basic jist is a sequence of events that looks like this:

      • mongod:
        2015-01-21T00:59:22.189+0000 I COMMAND  [conn15] command crapware_analysis.$cmd command: update { update: "etlq", updates: [ { q: { id: 7718 }, u: { $set: { lock: true } }, multi: false, upsert: false } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 141ms
        --lines omitted --
                2015-01-21T00:59:33.072+0000 F -        [conn15] Invalid access at address: 0x20
        --lines omitted --
                2015-01-21T00:59:33.586+0000 F -        [conn15] Got signal: 11 (Segmentation fault).
        
        0xf25749 0xf24df2 0xf2511e 0x7f5aa08e6340 0x12d11d3 0x13092ce 0x1309896 0x130756d 0x12c62cd 0x12df145 0x12ba500 0x12f4eb6 0xd47aed 0x8f073e 0x8f0f0c 0x98dd24 0x98f11a 0x9908b4 0x990fb5 0x99306d 0x9b31c4 0x9b4103 0x9b4bbb 0xb80ea5 0xa941fa 0x7e5320 0xee37d1 0x7f5aa08de182 0x7f5a9f9deefd
        ----- BEGIN BACKTRACE -----
        {"backtrace":[{"b":"400000","o":"B25749"},{"b":"400000","o":"B24DF2"},{"b":"400000","o":"B2511E"},{"b":"7F5AA08D6000","o":"10340"},{"b":"400000","o":"ED11D3"},{"b":"400000","o":"F092CE"},{"b":"400000","o":"F09896"},{"b":"400000","o":"F0756D"},{"b":"400000","o":"EC62CD"},{"b":"400000","o":"EDF145"},{"b":"400000","o":"EBA500"},{"b":"400000","o":"EF4EB6"},{"b":"400000","o":"947AED"},{"b":"400000","o":"4F073E"},{"b":"400000","o":"4F0F0C"},{"b":"400000","o":"58DD24"},{"b":"400000","o":"58F11A"},{"b":"400000","o":"5908B4"},{"b":"400000","o":"590FB5"},{"b":"400000","o":"59306D"},{"b":"400000","o":"5B31C4"},{"b":"400000","o":"5B4103"},{"b":"400000","o":"5B4BBB"},{"b":"400000","o":"780EA5"},{"b":"400000","o":"6941FA"},{"b":"400000","o":"3E5320"},{"b":"400000","o":"AE37D1"},{"b":"7F5AA08D6000","o":"8182"},{"b":"7F5A9F8E4000","o":"FAEFD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc5", "gitVersion" : "74b351de21c84438b12a83b28e155f5e69e3c1eb", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF863BF000", "elfType" : 3 }, { "b" : "7F5AA08D6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F5AA06CE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F5AA04CA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F5AA01C6000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F5A9FEC0000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F5A9FCAA000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F5A9F8E4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F5AA0AF4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
         mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf25749]
         mongod(+0xB24DF2) [0xf24df2]
         mongod(+0xB2511E) [0xf2511e]
         libpthread.so.0(+0x10340) [0x7f5aa08e6340]
         mongod(__wt_split_insert+0x813) [0x12d11d3]
         mongod(+0xF092CE) [0x13092ce]
         mongod(__wt_evict+0x96) [0x1309896]
         mongod(__wt_evict_page+0x2D) [0x130756d]
         mongod(__wt_page_in_func+0x64D) [0x12c62cd]
         mongod(__wt_row_search+0x9D5) [0x12df145]
         mongod(__wt_btcur_insert+0x300) [0x12ba500]
         mongod(+0xEF4EB6) [0x12f4eb6]
         mongod(_ZN5mongo21WiredTigerRecordStore12insertRecordEPNS_16OperationContextEPKcib+0xCD) [0xd47aed]
         mongod(_ZN5mongo10Collection15_insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x4E) [0x8f073e]
         mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x8C) [0x8f0f0c]
         mongod(_ZN5mongo18WriteBatchExecutor13execOneInsertEPNS0_16ExecInsertsStateEPPNS_16WriteErrorDetailE+0x214) [0x98dd24]
         mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x25A) [0x98f11a]
         mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x34) [0x9908b4]
         mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x395) [0x990fb5]
         mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x15D) [0x99306d]
         mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b31c4]
         mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9b4103]
         mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9b4bbb]
         mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x755) [0xb80ea5]
         mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xB0A) [0xa941fa]
         mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e5320]
         mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xee37d1]
         libpthread.so.0(+0x8182) [0x7f5aa08de182]
         libc.so.6(clone+0x6D) [0x7f5a9f9deefd]
        -----  END BACKTRACE  -----
        

      Results of syslog grep:

      /var/log/syslog.1:Jan 22 01:56:49 localhost kernel: [1227606.129859] init: mongod main process (32337) terminated with status 14
      

      Some details about my workload/deployment:

      • mongod runs on an Ubuntu LTS EC2 i2.xlarge instance without any other processes
      • db.stats() run after restart:
        {
                "db" : "crapware_analysis",
                "collections" : 11,
                "objects" : 159811819,
                "avgObjSize" : 469.8332837886039,
                "dataSize" : 75084911709,
                "storageSize" : 23971676160,
                "numExtents" : 0,
                "indexes" : 10,
                "indexSize" : 16404348928,
                "ok" : 1
        }
        
      • mongod.conf:
        dbpath=/opt/mongodb/wt
        logpath=/opt/mongodb/mongod.log
        logappend=true
        storageEngine = wiredTiger
        
      • Collection stats attached as "debug.out"
      • About 99% of the objects were produced in a workload that was running for about 6h, up to the point of the crash

            Assignee:
            Unassigned Unassigned
            Reporter:
            zaklee Zak Lee
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: