Details
-
Bug
-
Resolution: Duplicate
-
Critical - P2
-
None
-
2.8.0-rc5
-
None
-
ALL
-
- See above for detailed system information and database information; I can provide additional logs or information
- queue up workload as described
Description
I have an ETL workload that I have been testing a 2.8.0 WiredTiger single mongod deployment with. I have been able to reproduce this crash twice. As far as I can tell, there is no data loss after the crash. The basic jist is a sequence of events that looks like this:
- mongod:
2015-01-21T00:59:22.189+0000 I COMMAND [conn15] command crapware_analysis.$cmd command: update { update: "etlq", updates: [ { q: { id: 7718 }, u: { $set: { lock: true } }, multi: false, upsert: false } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:55 141ms--lines omitted --2015-01-21T00:59:33.072+0000 F - [conn15] Invalid access at address: 0x20--lines omitted --2015-01-21T00:59:33.586+0000 F - [conn15] Got signal: 11 (Segmentation fault).0xf25749 0xf24df2 0xf2511e 0x7f5aa08e6340 0x12d11d3 0x13092ce 0x1309896 0x130756d 0x12c62cd 0x12df145 0x12ba500 0x12f4eb6 0xd47aed 0x8f073e 0x8f0f0c 0x98dd24 0x98f11a 0x9908b4 0x990fb5 0x99306d 0x9b31c4 0x9b4103 0x9b4bbb 0xb80ea5 0xa941fa 0x7e5320 0xee37d1 0x7f5aa08de182 0x7f5a9f9deefd----- BEGIN BACKTRACE -----{"backtrace":[{"b":"400000","o":"B25749"},{"b":"400000","o":"B24DF2"},{"b":"400000","o":"B2511E"},{"b":"7F5AA08D6000","o":"10340"},{"b":"400000","o":"ED11D3"},{"b":"400000","o":"F092CE"},{"b":"400000","o":"F09896"},{"b":"400000","o":"F0756D"},{"b":"400000","o":"EC62CD"},{"b":"400000","o":"EDF145"},{"b":"400000","o":"EBA500"},{"b":"400000","o":"EF4EB6"},{"b":"400000","o":"947AED"},{"b":"400000","o":"4F073E"},{"b":"400000","o":"4F0F0C"},{"b":"400000","o":"58DD24"},{"b":"400000","o":"58F11A"},{"b":"400000","o":"5908B4"},{"b":"400000","o":"590FB5"},{"b":"400000","o":"59306D"},{"b":"400000","o":"5B31C4"},{"b":"400000","o":"5B4103"},{"b":"400000","o":"5B4BBB"},{"b":"400000","o":"780EA5"},{"b":"400000","o":"6941FA"},{"b":"400000","o":"3E5320"},{"b":"400000","o":"AE37D1"},{"b":"7F5AA08D6000","o":"8182"},{"b":"7F5A9F8E4000","o":"FAEFD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc5", "gitVersion" : "74b351de21c84438b12a83b28e155f5e69e3c1eb", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF863BF000", "elfType" : 3 }, { "b" : "7F5AA08D6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F5AA06CE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F5AA04CA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F5AA01C6000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F5A9FEC0000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F5A9FCAA000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F5A9F8E4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F5AA0AF4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf25749]mongod(+0xB24DF2) [0xf24df2]mongod(+0xB2511E) [0xf2511e]libpthread.so.0(+0x10340) [0x7f5aa08e6340]mongod(__wt_split_insert+0x813) [0x12d11d3]mongod(+0xF092CE) [0x13092ce]mongod(__wt_evict+0x96) [0x1309896]mongod(__wt_evict_page+0x2D) [0x130756d]mongod(__wt_page_in_func+0x64D) [0x12c62cd]mongod(__wt_row_search+0x9D5) [0x12df145]mongod(__wt_btcur_insert+0x300) [0x12ba500]mongod(+0xEF4EB6) [0x12f4eb6]mongod(_ZN5mongo21WiredTigerRecordStore12insertRecordEPNS_16OperationContextEPKcib+0xCD) [0xd47aed]mongod(_ZN5mongo10Collection15_insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x4E) [0x8f073e]mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x8C) [0x8f0f0c]mongod(_ZN5mongo18WriteBatchExecutor13execOneInsertEPNS0_16ExecInsertsStateEPPNS_16WriteErrorDetailE+0x214) [0x98dd24]mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x25A) [0x98f11a]mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x34) [0x9908b4]mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x395) [0x990fb5]mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x15D) [0x99306d]mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b31c4]mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9b4103]mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9b4bbb]mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x755) [0xb80ea5]mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xB0A) [0xa941fa]mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e5320]mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xee37d1]libpthread.so.0(+0x8182) [0x7f5aa08de182]libc.so.6(clone+0x6D) [0x7f5a9f9deefd]----- END BACKTRACE -----
Results of syslog grep:
/var/log/syslog.1:Jan 22 01:56:49 localhost kernel: [1227606.129859] init: mongod main process (32337) terminated with status 14
|
Some details about my workload/deployment:
- mongod runs on an Ubuntu LTS EC2 i2.xlarge instance without any other processes
- db.stats() run after restart:
{"db" : "crapware_analysis","collections" : 11,"objects" : 159811819,"avgObjSize" : 469.8332837886039,"dataSize" : 75084911709,"storageSize" : 23971676160,"numExtents" : 0,"indexes" : 10,"indexSize" : 16404348928,"ok" : 1} - mongod.conf:
dbpath=/opt/mongodb/wtlogpath=/opt/mongodb/mongod.loglogappend=truestorageEngine = wiredTiger - Collection stats attached as "debug.out"
- About 99% of the objects were produced in a workload that was running for about 6h, up to the point of the crash
Attachments
Issue Links
- duplicates
-
SERVER-16886 Segmentation fault when loading data into WiredTiger.
-
- Closed
-