Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17364

Idle cursors shouldn't hold storage engine resources

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • Quint Iteration 5, Quint Iteration 6

      Currently each idle ClientCursor holds on to it's original RecoveryUnit, so that getMores use the same session as the operation that created the cursor. This is a problem because they are relatively heavy-weight and there are a finite supply of them, at least for WiredTiger. We will change this to have getMores always use their own RU rather than swapping in the original. This requires some changes to the WT SDI::Cursor and RecordIterator implementations to be prepared for changes to the RU.

      ORIGINAL TITLE AND DESCRIPTION: Too many open WT_SESSIONs causes fassert

      Too many open cursors results in too many open sessions and an fassert. Probably we should just fail the cursor open in this case, not fassert.

      2015-02-24T11:48:37.426-0500 I -        [conn8] Invariant failure: ret resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 49
2015-02-24T11:48:37.426-0500 E STORAGE  [conn6] WiredTiger (12) [1424796517:426942][50894:0x11193d000], connection.open_session: only configured to support 20010 sessions (including 10 internal): Cannot allocate memory
2015-02-24T11:48:37.427-0500 I -        [conn6] Invariant failure: ret resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 49
2015-02-24T11:48:37.443-0500 I CONTROL  [conn4] 
 0x10e85e199 0x10e8124e0 0x10e7ffa65 0x10e690fd9 0x10e691ccd 0x10e68fdef 0x10e68be81 0x10e689d6c 0x10e30a1c8 0x10e4ed3f0 0x10e4ed2b9 0x10e4badd5 0x10e3bcbf6 0x10e1069b1 0x10e821f9c 0x10e891991 0x7fff8888f899 0x7fff8888f72a 0x7fff88893fc9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"10E0FB000","o":"763199"},{"b":"10E0FB000","o":"7174E0"},{"b":"10E0FB000","o":"704A65"},{"b":"10E0FB000","o":"595FD9"},{"b":"10E0FB000","o":"596CCD"},{"b":"10E0FB000","o":"594DEF"},{"b":"10E0FB000","o":"590E81"},{"b":"10E0FB000","o":"58ED6C"},{"b":"10E0FB000","o":"20F1C8"},{"b":"10E0FB000","o":"3F23F0"},{"b":"10E0FB000","o":"3F22B9"},{"b":"10E0FB000","o":"3BFDD5"},{"b":"10E0FB000","o":"2C1BF6"},{"b":"10E0FB000","o":"B9B1"},{"b":"10E0FB000","o":"726F9C"},{"b":"10E0FB000","o":"796991"},{"b":"7FFF8888E000","o":"1899"},{"b":"7FFF8888E000","o":"172A"},{"b":"7FFF8888E000","o":"5FC9"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9", "gitVersion" : "e6577bc37a2edba81b99146934cf7bad00c6e1b2", "uname" : { "sysname" : "Darwin", "release" : "13.4.0", "version" : "Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/bdlucas/mongodb/mongodb-osx-x86_64-3.0.0-rc9/bin/mongod", "machType" : 2, "b" : "10E0FB000", "buildId" : "E70F00A36A1938D48D63D6DDF7D85BDB" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8AC5A000", "buildId" : "70B235FCBCED367BBA6C67C299BAE7D9" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8AB46000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF85D3B000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8488B000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF871A8000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF87004000", "buildId" : "CF29DFF605893590834C82E2316612E8" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8FB5A000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF871B3000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF89668000", "buildId" : "41077DD7F9093B8A863E72AE304EDE13" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8C53B000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8F090000", "buildId" : "A40A0C7B321639B48AE0B5D3BAF1DA8A" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8A8F7000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF8EF5D000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8F6E2000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8F7E8000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8495D000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF85C52000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8A66F000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8AB99000", "buildId" : "3F8C6A0730463E88858FD9CEFC43A405" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF85BCC000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF9233A000", "buildId" : "9EDE872E2A9E3A788E1DAB790794A098" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF881E7000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF8B098000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8921D000", "buildId" : "8B1E1F1DA5CC3BAE8B1EABC84337A364" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF90AD0000", "buildId" : "9B34B4FEF5AD3F09A5F046AFF3571323" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF882AF000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8888E000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF8BDF6000", "buildId" : "0D0B13EA6B7A3AC8BE60B548543BEB77" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF89245000", "buildId" : "C588E082D94B35109F9A7AD83B3402DE" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8F6E0000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF892A7000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF863A9000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF8C075000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8A531000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF921D9000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF863A7000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10e85e199]
 mongod(_ZN5mongo10logContextEPKc+0x100) [0x10e8124e0]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x175) [0x10e7ffa65]
 mongod(_ZN5mongo17WiredTigerSessionC2EP15__wt_connectionii+0x99) [0x10e690fd9]
 mongod(_ZN5mongo22WiredTigerSessionCache10getSessionEv+0xAD) [0x10e691ccd]
 mongod(_ZN5mongo16WiredTigerCursorC2ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEybPNS_16OperationContextE+0x4F) [0x10e68fdef]
 mongod(_ZN5mongo21WiredTigerRecordStore8IteratorC2ERKS0_PNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionEb+0x61) [0x10e68be81]
 mongod(_ZNK5mongo21WiredTigerRecordStore11getIteratorEPNS_16OperationContextERKNS_8RecordIdERKNS_20CollectionScanParams9DirectionE+0x6C) [0x10e689d6c]
 mongod(_ZN5mongo14CollectionScan4workEPm+0x198) [0x10e30a1c8]
 mongod(_ZN5mongo12PlanExecutor18getNextSnapshottedEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0xA0) [0x10e4ed3f0]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0x10e4ed2b9]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0xA35) [0x10e4badd5]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xA36) [0x10e3bcbf6]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE1) [0x10e1069b1]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x32C) [0x10e821f9c]
 mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x10e891991]
 libsystem_pthread.dylib(_pthread_body+0x8A) [0x7fff8888f899]
 libsystem_pthread.dylib(_pthread_struct_init+0x0) [0x7fff8888f72a]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff88893fc9]
-----  END BACKTRACE  -----
2015-02-24T11:48:37.443-0500 I -        [conn4]

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            5 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated:
              Resolved: