[SERVER-16344] invariant failure: missing required lock in AutoGetCollectionForRead - MongoDB Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major - P3
Resolution: Cannot Reproduce
Affects Version/s: 2.8.0-rc1
Fix Version/s: None
Component/s: Concurrency, Storage
Labels:
- wiredtiger
Environment:

Hide
amazon m3.xlarge, ubuntu 14.04.
------------------------
mongod.conf
------------------------
systemLog:
   destination: file
   path: "/storage/logs/mongodb.log"
   logAppend: true
storage:
   dbPath: "/storage/data"
   engine: "wiredTiger"
   wiredTiger:
      engineConfig: "cache_size=12G"
      collectionConfig: "block_compressor=snappy"
   journal:
      enabled: true
   indexBuildRetry: false
net:
   port: 27017

Show
amazon m3.xlarge, ubuntu 14.04. ------------------------ mongod.conf ------------------------ systemLog:    destination: file    path: "/storage/logs/mongodb.log"    logAppend: true storage:    dbPath: "/storage/data"    engine: "wiredTiger"    wiredTiger:       engineConfig: "cache_size=12G"       collectionConfig: "block_compressor=snappy"    journal:       enabled: true    indexBuildRetry: false net:    port: 27017

Backwards Compatibility:
Fully Compatible
Operating System:
ALL

Description

I can't construct compound index (like {source:1, sourceId:1}) in foreground mode on a big collection (~213 millions of records, ~500gb of snappy compressed data). After 60-70% (depend on attempts I made) mongod crashes.

But I CAN build it in background mode without any problems.

Here is backtrace (sorry, I don't know how to collapse it in jira):

2014-11-27T22:48:24.011+0000 I -        [conn61]   Index Build: 151044900/213991103 70%

2014-11-27T22:48:27.023+0000 I -        [conn61]   Index Build: 151066000/213991103 70%

2014-11-27T22:48:30.015+0000 I -        [conn61]   Index Build: 151087600/213991103 70%

2014-11-27T22:48:30.514+0000 I -        [conn65] LockerId 983034 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack

 of the thread that has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an ov

erload.

2014-11-27T22:48:30.514+0000 I -        [conn65] Dumping LockManager @ 0x1b21958

2014-11-27T22:48:30.514+0000 I -        [conn65] Lock @ 0x3F9C940: {2305843009213693953: Global, 1}

GRANTED:

        LockRequest 982999 @ 0x6764700: Mode = IX; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

        LockRequest 983001 @ 0x3FDC300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

        LockRequest 983018 @ 0x518C300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

        LockRequest 983034 @ 0xDF89100: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

PENDING:

2014-11-27T22:48:30.514+0000 I -        [conn65] Lock @ 0x3F9BFE0: {7353707421137559428: Database, 436178393496477572}

GRANTED:

        LockRequest 982999 @ 0x6764700: Mode = X; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

2014-11-27T22:48:33.011+0000 I -        [conn61]   Index Build: 151109400/213991103 70%

2014-11-27T22:48:35.042+0000 I -        [clientcursormon] Invariant failure txn->lockState()->isDbLockedForMode(db, MODE_IS) src/mongo/db/catalog/database_holder.cpp 75

2014-11-27T22:48:35.051+0000 I CONTROL  [clientcursormon]

 0xf41389 0xee8d61 0xecdf82 0x902001 0x912023 0x9121ae 0x8f6703 0x915395 0xed0e49 0xf8e994 0x7f263e784182 0x7f263d884fbd

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"AE8D61"},{"b":"400000","o":"ACDF82"},{"b":"400000","o":"502001"},{"b":"400000","o":"512023"},{"b":"400000","o":"5121AE"},{"b":"400000","o":"4F6703"},{"b":"400000","o":"515395"},{"b":"400000","o":"AD0E49"},{"b":"400000","o":"B8E994"},{"b":"7F263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]

 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee8d61]

 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xecdf82]

 mongod(_ZNK5mongo14DatabaseHolder3getEPNS_16OperationContextERKNS_10StringDataE+0xD1) [0x902001]

 mongod(_ZN5mongo9AutoGetDbC1EPNS_16OperationContextERKNS_10StringDataENS_8LockModeE+0x43) [0x912023]

 mongod(_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKSs+0x6E) [0x9121ae]

 mongod(_ZN5mongo19GlobalCursorIdCache14timeoutCursorsEPNS_16OperationContextEi+0x113) [0x8f6703]

 mongod(_ZN5mongo19ClientCursorMonitor3runEv+0xA5) [0x915395]

 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x119) [0xed0e49]

 mongod(+0xB8E994) [0xf8e994]

 libpthread.so.0(+0x8182) [0x7f263e784182]

 libc.so.6(clone+0x6D) [0x7f263d884fbd]

-----  END BACKTRACE  -----

2014-11-27T22:48:35.051+0000 I -        [clientcursormon]

***aborting after invariant() failure

PENDING:

        LockRequest 983001 @ 0x3FDC300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

        LockRequest 983018 @ 0x518C300: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

        LockRequest 983034 @ 0xDF89100: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

2014-11-27T22:48:30.514+0000 I -        [conn65] Lock @ 0x3F9C160: {10974972495614759045: Collection, 1751600458759983237}

GRANTED:

        LockRequest 982999 @ 0x6764700: Mode = IX; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0;

PENDING:

2014-11-27T22:48:30.514+0000 I -        [conn65]

2014-11-27T22:48:30.519+0000 I -        [conn65]

 0xf41389 0x9a673a 0x9ae68c 0x99fcae 0x952282 0x9b67e4 0x9b7682 0x9b81f0 0xbb973c 0xa9db93 0x7e0290 0xefd9c1 0x7f263e784182 0x7f263d884fbd

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"5A673A"},{"b":"400000","o":"5AE68C"},{"b":"400000","o":"59FCAE"},{"b":"400000","o":"552282"},{"b":"400000","o":"5B67E4

"},{"b":"400000","o":"5B7682"},{"b":"400000","o":"5B81F0"},{"b":"400000","o":"7B973C"},{"b":"400000","o":"69DB93"},{"b":"400000","o":"3E0290"},{"b":"400000","o":"AFD9C1"},{"b":"7F

263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "s

ysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }

, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64

-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-li

nux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-

gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.

so.2", "elfType" : 3 } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]

 mongod(+0x5A673A) [0x9a673a]

 mongod(_ZN5mongo10LockerImplILb0EE12lockCompleteENS_10ResourceIdEjb+0x18C) [0x9ae68c]

 mongod(_ZN5mongo4Lock6DBLockC1EPNS_6LockerERKNS_10StringDataENS_8LockModeE+0xCE) [0x99fcae]

 mongod(_ZN5mongo16CmdListDatabases3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x262) [0x952282]

 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b67e4]

 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC62) [0x9b7682]

 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2D0) [0x9b81f0]

 mongod(_ZN5mongo11newRunQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_b+0x101C) [0xbb973c]

 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xBB3) [0xa9db93]

 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e0290]

 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x421) [0xefd9c1]

 libpthread.so.0(+0x8182) [0x7f263e784182]

 libc.so.6(clone+0x6D) [0x7f263d884fbd]

-----  END BACKTRACE  -----

2014-11-27T22:48:35.060+0000 F -        [clientcursormon] Got signal: 6 (Aborted).

 0xf41389 0xf40cf2 0xf41076 0x7f263d7c0c30 0x7f263d7c0bb9 0x7f263d7c3fc8 0xecdfc7 0x902001 0x912023 0x9121ae 0x8f6703 0x915395 0xed0e49 0xf8e994 0x7f263e784182 0x7f263d884fbd

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"B41389"},{"b":"400000","o":"B40CF2"},{"b":"400000","o":"B41076"},{"b":"7F263D78A000","o":"36C30"},{"b":"7F263D78A000","o":"36BB9"},{"b":"7F263D78A000","o":"39FC8"},{"b":"400000","o":"ACDFC7"},{"b":"400000","o":"502001"},{"b":"400000","o":"512023"},{"b":"400000","o":"5121AE"},{"b":"400000","o":"4F6703"},{"b":"400000","o":"515395"},{"b":"400000","o":"AD0E49"},{"b":"400000","o":"B8E994"},{"b":"7F263E77C000","o":"8182"},{"b":"7F263D78A000","o":"FAFBD"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1", "gitVersion" : "c8171e7f969519af8b87a43425ae291ee69a0191", "uname" : { "sysname" : "Linux", "release" : "3.13.0-36-generic", "version" : "#63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFFE663000", "elfType" : 3 }, { "b" : "7F263E77C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F263E574000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F263E370000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F263E06C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F263DD66000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F263DB50000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F263D78A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F263E99A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41389]

 mongod(+0xB40CF2) [0xf40cf2]

 mongod(+0xB41076) [0xf41076]

 libc.so.6(+0x36C30) [0x7f263d7c0c30]

 libc.so.6(gsignal+0x39) [0x7f263d7c0bb9]

 libc.so.6(abort+0x148) [0x7f263d7c3fc8]

 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xF7) [0xecdfc7]

 mongod(_ZNK5mongo14DatabaseHolder3getEPNS_16OperationContextERKNS_10StringDataE+0xD1) [0x902001]

 mongod(_ZN5mongo9AutoGetDbC1EPNS_16OperationContextERKNS_10StringDataENS_8LockModeE+0x43) [0x912023]

 mongod(_ZN5mongo24AutoGetCollectionForReadC2EPNS_16OperationContextERKSs+0x6E) [0x9121ae]

 mongod(_ZN5mongo19GlobalCursorIdCache14timeoutCursorsEPNS_16OperationContextEi+0x113) [0x8f6703]

 mongod(_ZN5mongo19ClientCursorMonitor3runEv+0xA5) [0x915395]

 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x119) [0xed0e49]

 mongod(+0xB8E994) [0xf8e994]

 libpthread.so.0(+0x8182) [0x7f263e784182]

 libc.so.6(clone+0x6D) [0x7f263d884fbd]

-----  END BACKTRACE  -----

Attachments

Issue Links

related to

SERVER-16055 Commands with empty string cause the server to terminate

Closed

Activity

People

Assignee:: Kaloian Manassiev

Reporter:: Dmitriy Selivanov

Participants:: Dmitriy Selivanov, Kaloian Manassiev

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: Nov 27 2014 11:05:09 PM UTC

Updated:: Jan 11 2015 08:36:33 PM UTC

Resolved:: Dec 15 2014 02:57:05 PM UTC