[SERVER-16993] mongod crashes with Got signal: 11 (Segmentation fault) Created: 22/Jan/15  Updated: 22/Jan/15  Resolved: 22/Jan/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 2.8.0-rc5
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Zak Lee Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File debug.out    
Issue Links:
Duplicate
duplicates SERVER-16886 Segmentation fault when loading data ... Closed
Operating System: ALL
Steps To Reproduce:
  • See above for detailed system information and database information; I can provide additional logs or information
  1. queue up workload as described
Participants:

 Description   

I have an ETL workload that I have been testing a 2.8.0 WiredTiger single mongod deployment with. I have been able to reproduce this crash twice. As far as I can tell, there is no data loss after the crash. The basic jist is a sequence of events that looks like this:

  • mongod:

    2015-01-21T00:59:22.189+0000 I COMMAND  [conn15] command crapware_analysis.$cmd command: update { update: "etlq", updates: [ { q: { id: 7718 }, u: { $set: { lock: true } }, multi: false, upsert: false } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 141ms
    --lines omitted --
            2015-01-21T00:59:33.072+0000 F -        [conn15] Invalid access at address: 0x20
    --lines omitted --
            2015-01-21T00:59:33.586+0000 F -        [conn15] Got signal: 11 (Segmentation fault).
     
    0xf25749 0xf24df2 0xf2511e 0x7f5aa08e6340 0x12d11d3 0x13092ce 0x1309896 0x130756d 0x12c62cd 0x12df145 0x12ba500 0x12f4eb6 0xd47aed 0x8f073e 0x8f0f0c 0x98dd24 0x98f11a 0x9908b4 0x990fb5 0x99306d 0x9b31c4 0x9b4103 0x9b4bbb 0xb80ea5 0xa941fa 0x7e5320 0xee37d1 0x7f5aa08de182 0x7f5a9f9deefd
    ----- BEGIN BACKTRACE -----
    {"backtrace":[{"b":"400000","o":"B25749"},{"b":"400000","o":"B24DF2"},{"b":"400000","o":"B2511E"},{"b":"7F5AA08D6000","o":"10340"},{"b":"400000","o":"ED11D3"},{"b":"400000","o":"F092CE"},{"b":"400000","o":"F09896"},{"b":"400000","o":"F0756D"},{"b":"400000","o":"EC62CD"},{"b":"400000","o":"EDF145"},{"b":"400000","o":"EBA500"},{"b":"400000","o":"EF4EB6"},{"b":"400000","o":"947AED"},{"b":"400000","o":"4F073E"},{"b":"400000","o":"4F0F0C"},{"b":"400000","o":"58DD24"},{"b":"400000","o":"58F11A"},{"b":"400000","o":"5908B4"},{"b":"400000","o":"590FB5"},{"b":"400000","o":"59306D"},{"b":"400000","o":"5B31C4"},{"b":"400000","o":"5B4103"},{"b":"400000","o":"5B4BBB"},{"b":"400000","o":"780EA5"},{"b":"400000","o":"6941FA"},{"b":"400000","o":"3E5320"},{"b":"400000","o":"AE37D1"},{"b":"7F5AA08D6000","o":"8182"},{"b":"7F5A9F8E4000","o":"FAEFD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc5", "gitVersion" : "74b351de21c84438b12a83b28e155f5e69e3c1eb", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF863BF000", "elfType" : 3 }, { "b" : "7F5AA08D6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F5AA06CE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F5AA04CA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F5AA01C6000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F5A9FEC0000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F5A9FCAA000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F5A9F8E4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F5AA0AF4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
     mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf25749]
     mongod(+0xB24DF2) [0xf24df2]
     mongod(+0xB2511E) [0xf2511e]
     libpthread.so.0(+0x10340) [0x7f5aa08e6340]
     mongod(__wt_split_insert+0x813) [0x12d11d3]
     mongod(+0xF092CE) [0x13092ce]
     mongod(__wt_evict+0x96) [0x1309896]
     mongod(__wt_evict_page+0x2D) [0x130756d]
     mongod(__wt_page_in_func+0x64D) [0x12c62cd]
     mongod(__wt_row_search+0x9D5) [0x12df145]
     mongod(__wt_btcur_insert+0x300) [0x12ba500]
     mongod(+0xEF4EB6) [0x12f4eb6]
     mongod(_ZN5mongo21WiredTigerRecordStore12insertRecordEPNS_16OperationContextEPKcib+0xCD) [0xd47aed]
     mongod(_ZN5mongo10Collection15_insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x4E) [0x8f073e]
     mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x8C) [0x8f0f0c]
     mongod(_ZN5mongo18WriteBatchExecutor13execOneInsertEPNS0_16ExecInsertsStateEPPNS_16WriteErrorDetailE+0x214) [0x98dd24]
     mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x25A) [0x98f11a]
     mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x34) [0x9908b4]
     mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x395) [0x990fb5]
     mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x15D) [0x99306d]
     mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b31c4]
     mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9b4103]
     mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9b4bbb]
     mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x755) [0xb80ea5]
     mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xB0A) [0xa941fa]
     mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e5320]
     mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xee37d1]
     libpthread.so.0(+0x8182) [0x7f5aa08de182]
     libc.so.6(clone+0x6D) [0x7f5a9f9deefd]
    -----  END BACKTRACE  -----

Results of syslog grep:

/var/log/syslog.1:Jan 22 01:56:49 localhost kernel: [1227606.129859] init: mongod main process (32337) terminated with status 14

Some details about my workload/deployment:

  • mongod runs on an Ubuntu LTS EC2 i2.xlarge instance without any other processes
  • db.stats() run after restart:

    {
            "db" : "crapware_analysis",
            "collections" : 11,
            "objects" : 159811819,
            "avgObjSize" : 469.8332837886039,
            "dataSize" : 75084911709,
            "storageSize" : 23971676160,
            "numExtents" : 0,
            "indexes" : 10,
            "indexSize" : 16404348928,
            "ok" : 1
    }

  • mongod.conf:

    dbpath=/opt/mongodb/wt
    logpath=/opt/mongodb/mongod.log
    logappend=true
    storageEngine = wiredTiger

  • Collection stats attached as "debug.out"
  • About 99% of the objects were produced in a workload that was running for about 6h, up to the point of the crash


 Comments   
Comment by Ramon Fernandez Marina [ 22/Jan/15 ]

Same stack trace as SERVER-16886, so closing as a duplicate.

SERVER-16886 has been fixed in master and will be part of the next release candidate.

Comment by Ramon Fernandez Marina [ 22/Jan/15 ]

Thanks for your report zaklee. This crash was previously reported in SERVER-16929 (see the stack trace here, which itself is a duplicate of SERVER-16886.

Regards,
Ramón.

Generated at Thu Feb 08 03:42:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.