Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19263

mongod hangs on db.shutdownServer() if fsyncLock()

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.0.3, 3.0.4, 3.1.5
    • Fix Version/s: None
    • Component/s: Stability
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      boot up mongod

      db.fsyncLock()
      db.adminCommand( {shutdown:1}) 
      

      system hangs

      Show
      boot up mongod db.fsyncLock() db.adminCommand( {shutdown:1}) system hangs

      Description

      While doing some testing on replica set's I've found an odd behaviour.
      mongod will hang indefinitely (and only reacts to kill -9) if one tries to db.shutdownServer() while having db.fsyncLock() active.

      After a kill -9 on the mongod process, the MongoDB shell throws the following backtrace:

      nair(mongod-3.0.3)[SECONDARY] admin> db.adminCommand( {shutdown:1})
      ^C2015-07-02T15:18:50.646+0200 I NETWORK  Socket recv() errno:54 Connection reset by peer 127.0.0.1:30002
      2015-07-02T15:18:50.648+0200 I NETWORK  SocketException: remote: 127.0.0.1:30002 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:30002]
      2015-07-02T15:18:50.648+0200 I NETWORK  DBClientCursor::init call() failed
      2015-07-02T15:18:50.648+0200 F -        terminate() called. An exception is active; attempting to gather more information
      2015-07-02T15:18:50.667+0200 F -        DBException::toString(): 10276 DBClientBase::findN: transport error: 127.0.0.1:30002 ns: admin.$cmd.sys.inprog query: {}
      Actual exception type: mongo::UserException
       
       0x10446c30a 0x10446ba02 0x7fff89cc81d1 0x7fff89cc7c5b 0x10440a228 0x10440a02a 0x1042f3672 0x1042f3770 0x1043fd199 0x1042a718d 0x1042a71d9 0x7fff86bc95aa 0x7fb10d81fd10 0x104425f4d 0x104426d02 0x1042fa8e6 0x10430de93 0x1042f5903 0x1042fc444 0x1043cd823 0x1043bfee2 0x3181f0289159
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"1042A5000","o":"1C730A"},{"b":"1042A5000","o":"1C6A02"},{"b":"7FFF89CA4000","o":"241D1"},{"b":"7FFF89CA4000","o":"23C5B"},{"b":"1042A5000","o":"165228"},{"b":"1042A5000","o":"16502A"},{"b":"1042A5000","o":"4E672"},{"b":"1042A5000","o":"4E770"},{"b":"1042A5000","o":"158199"},{"b":"1042A5000","o":"218D"},{"b":"1042A5000","o":"21D9"},{"b":"7FFF86BC6000","o":"35AA"},{"b":"0","o":"7FB10D81FD10"},{"b":"1042A5000","o":"180F4D"},{"b":"1042A5000","o":"181D02"},{"b":"1042A5000","o":"558E6"},{"b":"1042A5000","o":"68E93"},{"b":"1042A5000","o":"50903"},{"b":"1042A5000","o":"57444"},{"b":"1042A5000","o":"128823"},{"b":"1042A5000","o":"11AEE2"},{"b":"0","o":"3181F0289159"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Darwin", "release" : "13.4.0", "version" : "Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/norberto/bin/mongo", "machType" : 2, "b" : "1042A5000", "buildId" : "7CBFA011798A3C2F92CB8C2332A3206B" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8EF74000", "buildId" : "E303F2F8A8CF3DF384B3F2D0EE41CCF6" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF86440000", "buildId" : "4F68DFC5207739A8A449CAC5FDEE7BDE" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF8ACAD000", "buildId" : "BDC1E65B72A13DA3A57CB23159CAAD0B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8B712000", "buildId" : "8C4F0CA0389C3EDCB155E62DD2187E1D" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF84C2C000", "buildId" : "4CD916B21B17362AB403EF24A1DAC141" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF8ED55000", "buildId" : "CF29DFF605893590834C82E2316612E8" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF87F80000", "buildId" : "F3973C2814B63006BB2B00DD7F09ABC7" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF84CD5000", "buildId" : "C4E4A18D3C3B3C9C8709A4270D998DE7" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF859BF000", "buildId" : "7C9EC3B7DDE333FF953F4067C743951D" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF84CF0000", "buildId" : "3AA8D85DCF003BD3A5A0E28E1A32A6D8" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8A7C4000", "buildId" : "A40A0C7B321639B48AE0B5D3BAF1DA8A" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8CD36000", "buildId" : "1D2910DFC0363A82A3FD44FF73B5FF9B" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF83F5A000", "buildId" : "7A1A2BCBC03D3A25BFA43E569B2D2C38" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF89C3C000", "buildId" : "3543F917928E3DB2A2F47AB73B4970EF" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF91059000", "buildId" : "655FB34352CF3E2FB14DBEBF5AAEF94D" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8BCC4000", "buildId" : "FB856CD12AEA39078E9B1E54B6827F82" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF85AAB000", "buildId" : "6FD3A4004BB23B95B90CBE6E9D0D78FA" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8B5C1000", "buildId" : "4998CB6A9D54390A9F575D1AC53C135C" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8FB1F000", "buildId" : "17B03FFD92C532829981EBB28B456207" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF86935000", "buildId" : "7D41A156D2853849A2C3C04ADE797D98" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF85A8E000", "buildId" : "8116098DB3F13E50A934576DD6369234" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8807E000", "buildId" : "B7F0E2E4277733FCA787D6430B630D54" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF85975000", "buildId" : "A695B4E438E9332EA77229D31E3F1385" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF86FA4000", "buildId" : "0D630D53C7723EC58257EFB0ACCE3153" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8FDE2000", "buildId" : "9B34B4FEF5AD3F09A5F046AFF3571323" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF86BC6000", "buildId" : "3C3D3DA832B9324398ECD89B9A1670B3" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF91808000", "buildId" : "AB498556B555310E9041F67EC9E00E2C" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF83E02000", "buildId" : "0C93EB2373643670B511212A7A524695" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF8C46A000", "buildId" : "4E51D5B092A03D0DB90E495A1ED3E391" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF88C77000", "buildId" : "62682455186236FE8A047A6B91256438" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF8970C000", "buildId" : "78DCC3582FC1302EB3950155B47CB547" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF859C3000", "buildId" : "AB40CD57F4543FD4B41563B3C0D5C624" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF84A7E000", "buildId" : "AD7FD984271E30F4A3616B20319EC73B" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8EED1000", "buildId" : "F45C36E8B6063886B5B1B6745E757CA8" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF89CA4000", "buildId" : "21A807D367323455B77F743E9F916DF0" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF892C3000", "buildId" : "4CDB0F7BC0AF3424BC39495696F0DB1E" } ] }}
       mongo(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x3A) [0x10446c30a]
       mongo(_ZN5mongo12_GLOBAL__N_111myTerminateEv+0x282) [0x10446ba02]
       libc++abi.dylib(_ZSt11__terminatePFvvE+0x8) [0x7fff89cc81d1]
       libc++abi.dylib(_ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception+0x0) [0x7fff89cc7c5b]
       mongo(_ZN5mongo9uassertedEiPKc+0x1F8) [0x10440a228]
       mongo(_ZN5mongo9uassertedEiRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE+0x1A) [0x10440a02a]
       mongo(_ZN5mongo17DBClientInterface5findNERNSt3__16vectorINS_7BSONObjENS1_9allocatorIS3_EEEERKNS1_12basic_stringIcNS1_11char_traitsIcEENS4_IcEEEENS_5QueryEiiPKS3_i+0x632) [0x1042f3672]
       mongo(_ZN5mongo17DBClientInterface7findOneERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEERKNS_5QueryEPKNS_7BSONObjEi+0x60) [0x1042f3770]
       mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1D9) [0x1043fd199]
       mongo(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x6D) [0x1042a718d]
       mongo(_Z10quitNicelyi+0x9) [0x1042a71d9]
       libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff86bc95aa]
       ??? [0x7fb10d81fd10]
       mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x7D) [0x104425f4d]
       mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x22) [0x104426d02]
       mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE+0x46) [0x1042fa8e6]
       mongo(_ZN5mongo14DBClientCursor4initEv+0x93) [0x10430de93]
       mongo(_ZN5mongo12DBClientBase5queryERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS_5QueryEiiPKNS_7BSONObjEii+0xA3) [0x1042f5903]
       mongo(_ZN5mongo18DBClientConnection5queryERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS_5QueryEiiPKNS_7BSONObjEii+0x84) [0x1042fc444]
       mongo(_ZN5mongo9mongoFindEPNS_7V8ScopeERKN2v89ArgumentsE+0x2C3) [0x1043cd823]
       mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0x52) [0x1043bfee2]
       ??? [0x3181f0289159]
      -----  END BACKTRACE  -----
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              norberto.leite Norberto Fernando Rocha Leite (Inactive)
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: