Details
-
Bug
-
Status: Closed
-
Major - P3
-
Resolution: Cannot Reproduce
-
2.8.0-rc1
-
None
-
amazon m3.xlarge, ubuntu 14.04.
------------------------
mongod.conf
------------------------
systemLog:
destination: file
path: "/storage/logs/mongodb.log"
logAppend: true
storage:
dbPath: "/storage/data"
engine: "wiredTiger"
wiredTiger:
engineConfig: "cache_size=12G"
collectionConfig: "block_compressor=snappy"
journal:
enabled: true
indexBuildRetry: false
net:
port: 27017
amazon m3.xlarge, ubuntu 14.04. ------------------------ mongod.conf ------------------------ systemLog: destination: file path: "/storage/logs/mongodb.log" logAppend: true storage: dbPath: "/storage/data" engine: "wiredTiger" wiredTiger: engineConfig: "cache_size=12G" collectionConfig: "block_compressor=snappy" journal: enabled: true indexBuildRetry: false net: port: 27017
-
Fully Compatible
-
ALL
Description
I can't construct compound index (like {source:1, sourceId:1}) in foreground mode on a big collection (~213 millions of records, ~500gb of snappy compressed data). After 60-70% (depend on attempts I made) mongod crashes.
But I CAN build it in background mode without any problems.
Here is backtrace (sorry, I don't know how to collapse it in jira):
2014-11-27T22:48:24.011+0000 I - [conn61] Index Build: 151044900/213991103 70%
|
2014-11-27T22:48:27.023+0000 I - [conn61] Index Build: 151066000/213991103 70%
|
2014-11-27T22:48:30.015+0000 I - [conn61] Index Build: 151087600/213991103 70%
|
2014-11-27T22:48:30.514+0000 I - [conn65] LockerId 983034 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack
|
of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an ov
|
erload.
|
2014-11-27T22:48:30.514+0000 I - [conn65] Dumping LockManager @ 0x1b21958
|
2014-11-27T22:48:30.514+0000 I - [conn65] Lock @ 0x3F9C940: {2305843009213693953: Global, 1}
|
GRANTED:
|
LockRequest 982999 @ 0x6764700: Mode = IX; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
LockRequest 983001 @ 0x3FDC300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
LockRequest 983018 @ 0x518C300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
LockRequest 983034 @ 0xDF89100: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
|
PENDING:
|
2014-11-27T22:48:30.514+0000 I - [conn65] Lock @ 0x3F9BFE0: {7353707421137559428: Database, 436178393496477572}
|
GRANTED:
|
LockRequest 982999 @ 0x6764700: Mode = X; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
|
2014-11-27T22:48:33.011+0000 I - [conn61] Index Build: 151109400/213991103 70%
|
2014-11-27T22:48:35.042+0000 I - [clientcursormon] Invariant failure txn->lockState()->isDbLockedForMode(db, MODE_IS) src/mongo/db/catalog/database_holder.cpp 75
|
2014-11-27T22:48:35.051+0000 I CONTROL [clientcursormon]
|
0xf41389 0xee8d61 0xecdf82 0x902001 0x912023 0x9121ae 0x8f6703 0x915395 0xed0e49 0xf8e994 0x7f263e784182 0x7f263d884fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"AE8D61"},{"b":"400000","o":"ACDF82"},{"b":"400000","o":"502001"},{"b":"400000","o":"512023"},{"b":"400000","o":"5121AE"},{"b":"400000","o":"4F6703"},{"b":"400000","o":"515395"},{"b":"400000","o":"AD0E49"},{"b":"400000","o":"B8E994"},{"b":"7F263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]
|
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee8d61]
|
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xecdf82]
|
mongod(_ZNK5mongo14DatabaseHolder3getEPNS_16OperationContextERKNS_10StringDataE+0xD1) [0x902001]
|
mongod(_ZN5mongo9AutoGetDbC1EPNS_16OperationContextERKNS_10StringDataENS_8LockModeE+0x43) [0x912023]
|
mongod(_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKSs+0x6E) [0x9121ae]
|
mongod(_ZN5mongo19GlobalCursorIdCache14timeoutCursorsEPNS_16OperationContextEi+0x113) [0x8f6703]
|
mongod(_ZN5mongo19ClientCursorMonitor3runEv+0xA5) [0x915395]
|
mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x119) [0xed0e49]
|
mongod(+0xB8E994) [0xf8e994]
|
libpthread.so.0(+0x8182) [0x7f263e784182]
|
libc.so.6(clone+0x6D) [0x7f263d884fbd]
|
----- END BACKTRACE -----
|
2014-11-27T22:48:35.051+0000 I - [clientcursormon]
|
|
***aborting after invariant() failure
|
PENDING:
|
LockRequest 983001 @ 0x3FDC300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
LockRequest 983018 @ 0x518C300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
LockRequest 983034 @ 0xDF89100: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
2014-11-27T22:48:30.514+0000 I - [conn65] Lock @ 0x3F9C160: {10974972495614759045: Collection, 1751600458759983237}
|
GRANTED:
|
LockRequest 982999 @ 0x6764700: Mode = IX; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;
|
|
PENDING:
|
2014-11-27T22:48:30.514+0000 I - [conn65]
|
2014-11-27T22:48:30.519+0000 I - [conn65]
|
0xf41389 0x9a673a 0x9ae68c 0x99fcae 0x952282 0x9b67e4 0x9b7682 0x9b81f0 0xbb973c 0xa9db93 0x7e0290 0xefd9c1 0x7f263e784182 0x7f263d884fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"5A673A"},{"b":"400000","o":"5AE68C"},{"b":"400000","o":"59FCAE"},{"b":"400000","o":"552282"},{"b":"400000","o":"5B67E4
|
"},{"b":"400000","o":"5B7682"},{"b":"400000","o":"5B81F0"},{"b":"400000","o":"7B973C"},{"b":"400000","o":"69DB93"},{"b":"400000","o":"3E0290"},{"b":"400000","o":"AFD9C1"},{"b":"7F
|
263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "s
|
ysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }
|
, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64
|
-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-li
|
nux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-
|
gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.
|
so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]
|
mongod(+0x5A673A) [0x9a673a]
|
mongod(_ZN5mongo10LockerImplILb0EE12lockCompleteENS_10ResourceIdEjb+0x18C) [0x9ae68c]
|
mongod(_ZN5mongo4Lock6DBLockC1EPNS_6LockerERKNS_10StringDataENS_8LockModeE+0xCE) [0x99fcae]
|
mongod(_ZN5mongo16CmdListDatabases3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x262) [0x952282]
|
mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b67e4]
|
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC62) [0x9b7682]
|
mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2D0) [0x9b81f0]
|
mongod(_ZN5mongo11newRunQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_b+0x101C) [0xbb973c]
|
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xBB3) [0xa9db93]
|
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e0290]
|
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x421) [0xefd9c1]
|
libpthread.so.0(+0x8182) [0x7f263e784182]
|
libc.so.6(clone+0x6D) [0x7f263d884fbd]
|
----- END BACKTRACE -----
|
|
2014-11-27T22:48:35.060+0000 F - [clientcursormon] Got signal: 6 (Aborted).
|
|
0xf41389 0xf40cf2 0xf41076 0x7f263d7c0c30 0x7f263d7c0bb9 0x7f263d7c3fc8 0xecdfc7 0x902001 0x912023 0x9121ae 0x8f6703 0x915395 0xed0e49 0xf8e994 0x7f263e784182 0x7f263d884fbd
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"B40CF2"},{"b":"400000","o":"B41076"},{"b":"7F263D78A000","o":"36C30"},{"b":"7F263D78A000","o":"36BB9"},{"b":"7F263D78A000","o":"39FC8"},{"b":"400000","o":"ACDFC7"},{"b":"400000","o":"502001"},{"b":"400000","o":"512023"},{"b":"400000","o":"5121AE"},{"b":"400000","o":"4F6703"},{"b":"400000","o":"515395"},{"b":"400000","o":"AD0E49"},{"b":"400000","o":"B8E994"},{"b":"7F263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]
|
mongod(+0xB40CF2) [0xf40cf2]
|
mongod(+0xB41076) [0xf41076]
|
libc.so.6(+0x36C30) [0x7f263d7c0c30]
|
libc.so.6(gsignal+0x39) [0x7f263d7c0bb9]
|
libc.so.6(abort+0x148) [0x7f263d7c3fc8]
|
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xF7) [0xecdfc7]
|
mongod(_ZNK5mongo14DatabaseHolder3getEPNS_16OperationContextERKNS_10StringDataE+0xD1) [0x902001]
|
mongod(_ZN5mongo9AutoGetDbC1EPNS_16OperationContextERKNS_10StringDataENS_8LockModeE+0x43) [0x912023]
|
mongod(_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKSs+0x6E) [0x9121ae]
|
mongod(_ZN5mongo19GlobalCursorIdCache14timeoutCursorsEPNS_16OperationContextEi+0x113) [0x8f6703]
|
mongod(_ZN5mongo19ClientCursorMonitor3runEv+0xA5) [0x915395]
|
mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x119) [0xed0e49]
|
mongod(+0xB8E994) [0xf8e994]
|
libpthread.so.0(+0x8182) [0x7f263e784182]
|
libc.so.6(clone+0x6D) [0x7f263d884fbd]
|
----- END BACKTRACE -----
|
Attachments
Issue Links
- related to
-
SERVER-16055 Commands with empty string cause the server to terminate
-
- Closed
-