Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16558

Backtraces with delays (5 sec in wiredtiger_open) when trying to start server when it's already running

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Stability
    • None
    • ALL

      Trying to start mongod with the same config results in the following backtrace, if the process is already running:

      2014-12-15T17:13:45.212-0500 I CONTROL  ***** SERVER RESTARTED *****
      2014-12-15T17:13:45.222-0500 I CONTROL  [initandlisten] MongoDB starting : pid=99177 port=27019 dbpath=/tmp/fffx/db 64-bit host=AD-MAC10G.local
      2014-12-15T17:13:45.222-0500 I CONTROL  [initandlisten] db version v2.8.0-rc2
      2014-12-15T17:13:45.222-0500 I CONTROL  [initandlisten] git version: f77869e01a6ed235695a46fad441c0dff15202f6
      2014-12-15T17:13:45.223-0500 I CONTROL  [initandlisten] build info: Darwin mci-osx108-8.build.10gen.cc 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
      2014-12-15T17:13:45.223-0500 I CONTROL  [initandlisten] allocator: system
      2014-12-15T17:13:45.223-0500 I CONTROL  [initandlisten] options: { config: "config.yaml", cpu: true, net: { http: { RESTInterfaceEnabled: true, enabled: true }, port: 27019 }, operationProfiling: { mode: "slowOp" }, processManagement: { fork: true }, replication: { replSetName: "replset" }, setParameter: { internalQueryExecMaxBlockingSortBytes: "1073741824" }, storage: { dbPath: "/tmp/fffx/db", engine: "wiredtiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/tmp/fffx/log" } }
      2014-12-15T17:13:45.223-0500 E NETWORK  [initandlisten] listen(): bind() failed errno:48 Address already in use for socket: 0.0.0.0:27019
      2014-12-15T17:13:45.223-0500 E NETWORK  [initandlisten]   addr already in use
      2014-12-15T17:13:45.223-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,extensions=[local=(entry=index_collator_extension)],statistics=(all),log=(enabled=true,archive=true,path=journal),checkpoint=(wait=60,log_size=2GB),
      2014-12-15T17:13:50.232-0500 E STORAGE  [initandlisten] WiredTiger (16) [1418681630:232677][99177:0x7fff77e0c180], wiredtiger_open: WiredTiger database is already being managed by another process: Resource busy
      2014-12-15T17:13:50.233-0500 I -        [initandlisten] Invariant failure ret UnknownError 16: Resource busy src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 154
      2014-12-15T17:13:50.236-0500 I CONTROL  [initandlisten]
       0x1034cc839 0x103480900 0x10346eb21 0x1032fe5c5 0x1032fda9a 0x10302c307 0x102d9f075 0x102d9de93 0x102da2d1c 0x102d9de74 0x4
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"102D9D000","o":"72F839"},{"b":"102D9D000","o":"6E3900"},{"b":"102D9D000","o":"6D1B21"},{"b":"102D9D000","o":"5615C5"},{"b":"102D9D000","o":"560A9A"},{"b":"102D9D000","o":"28F307"},{"b":"102D9D000","o":"2075"},{"b":"102D9D000","o":"E93"},{"b":"102D9D000","o":"5D1C"},{"b":"102D9D000","o":"E74"},{"b":"0","o":"4"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc2", "gitVersion" : "f77869e01a6ed235695a46fad441c0dff15202f6", "uname" : { "sysname" : "Darwin", "release" : "12.3.0", "version" : "Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongod", "machType" : 2, "b" : "102D9D000", "buildId" : "5E5E1684499B308383920314F3A9D5EA" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF86A79000", "buildId" : "FF25248A574C32DB952FB948C389B2A4" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF89B59000", "buildId" : "20E31B9019B93C2AA9EB474E08F9FE05" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF86E31000", "buildId" : "65187C6E3FBF3EB8A1AA389445E2984D" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8A79C000", "buildId" : "BAAFE0C9BB863CA788C0E3CBA98DA06F" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8E976000", "buildId" : "08F8731D596139F1AD004590321D24A9" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF8E96E000", "buildId" : "876573D0E9073566A108577EAD1B6182" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF93855000", "buildId" : "D26996BFFC5739EB8829F63585561E09" }, { "path" : "/usr/lib/system/libdnsinfo.dylib", "machType" : 6, "b" : "7FFF8E8D0000", "buildId" : "14202FFBC3CA3FCC94B014611BF8692D" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8A81A000", "buildId" : "F59367C9C110382BA6959035A6DD387E" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8DC14000", "buildId" : "CC9E3394BE16397F926BE579B60EE429" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF90E1A000", "buildId" : "2F71CAF86524329EAC56C506658B4C0C" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8E7CD000", "buildId" : "BF332AD9E89F387E92A46E1AB74BD4D9" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF93464000", "buildId" : "4BE2E642A14F340AB4825BD2AEFD9C24" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8DB3F000", "buildId" : "6763BC8E18B83AD98FFAB43713A7264F" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8EB08000", "buildId" : "D92DCBC3541C37BDAADEACC75A0C59C8" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8FB79000", "buildId" : "4C9EB006FE1F3F8F8074DFD94CF2CE7B" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF86A7B000", "buildId" : "616FC901151E38BFB2C424A351C5FAAD" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF887CE000", "buildId" : "4FFCA2427F04365F87A6D4EFB89503C1" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8F915000", "buildId" : "5A961E2ACFB8362BBC43122704AEB047" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF88805000", "buildId" : "B434BE5C25AB3EBDBAA75304B34E3441" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF92ED8000", "buildId" : "0D99F24E56FE380FB81B4A4C630EE587" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8EB0A000", "buildId" : "C49275CC835A3207AFBA8C01374927B6" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF8FC47000", "buildId" : "6838A6FD86263356BB4FBB4787216207" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF9386B000", "buildId" : "92805328CD3634FF9436571AB0485072" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF9384E000", "buildId" : "21703D362DAB3D8B8442EAAB23C060D3" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF939B8000", "buildId" : "BBE558BD5E5535E489ED1AA6B056D05A" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8DB47000", "buildId" : "CE0C29A3C420339BADAA52F4683233CC" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF910B4000", "buildId" : "90D31928F48D3E37874F220A51FD9E37" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8C737000", "buildId" : "73CDC48216E33FC79BB4FBA2DA44DBC2" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8EA05000", "buildId" : "D86169F39F31377A9AF3DB17142052E4" } ] }}
       mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1034cc839]
       mongod(_ZN5mongo10logContextEPKc+0x100) [0x103480900]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x171) [0x10346eb21]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES9_b+0x775) [0x1032fe5c5]
       mongod(_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsE+0x3A) [0x1032fda9a]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE+0x77) [0x10302c307]
       mongod(_ZN5mongoL14_initAndListenEi+0xC95) [0x102d9f075]
       mongod(_ZN5mongo13initAndListenEi+0x13) [0x102d9de93]
       mongod(main+0x3BC) [0x102da2d1c]
       mongod(start+0x34) [0x102d9de74]
       ??? [0x4]
      -----  END BACKTRACE  -----
      2014-12-15T17:13:50.236-0500 I -        [initandlisten]
      
      ***aborting after invariant() failure
      
      
      2014-12-15T17:13:50.239-0500 F -        [initandlisten] Got signal: 6 (Abort trap: 6).
      
       0x1034cc839 0x1034cc276 0x7fff8fb7b94a 0x5f 0x7fff8fbd2dce 0x10346ebfe 0x1032fe5c5 0x1032fda9a 0x10302c307 0x102d9f075 0x102d9de93 0x102da2d1c 0x102d9de74 0x4
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"102D9D000","o":"72F839"},{"b":"102D9D000","o":"72F276"},{"b":"7FFF8FB79000","o":"294A"},{"b":"0","o":"5F"},{"b":"7FFF8FB79000","o":"59DCE"},{"b":"102D9D000","o":"6D1BFE"},{"b":"102D9D000","o":"5615C5"},{"b":"102D9D000","o":"560A9A"},{"b":"102D9D000","o":"28F307"},{"b":"102D9D000","o":"2075"},{"b":"102D9D000","o":"E93"},{"b":"102D9D000","o":"5D1C"},{"b":"102D9D000","o":"E74"},{"b":"0","o":"4"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc2", "gitVersion" : "f77869e01a6ed235695a46fad441c0dff15202f6", "uname" : { "sysname" : "Darwin", "release" : "12.3.0", "version" : "Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongod", "machType" : 2, "b" : "102D9D000", "buildId" : "5E5E1684499B308383920314F3A9D5EA" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF86A79000", "buildId" : "FF25248A574C32DB952FB948C389B2A4" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF89B59000", "buildId" : "20E31B9019B93C2AA9EB474E08F9FE05" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF86E31000", "buildId" : "65187C6E3FBF3EB8A1AA389445E2984D" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8A79C000", "buildId" : "BAAFE0C9BB863CA788C0E3CBA98DA06F" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8E976000", "buildId" : "08F8731D596139F1AD004590321D24A9" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF8E96E000", "buildId" : "876573D0E9073566A108577EAD1B6182" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF93855000", "buildId" : "D26996BFFC5739EB8829F63585561E09" }, { "path" : "/usr/lib/system/libdnsinfo.dylib", "machType" : 6, "b" : "7FFF8E8D0000", "buildId" : "14202FFBC3CA3FCC94B014611BF8692D" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8A81A000", "buildId" : "F59367C9C110382BA6959035A6DD387E" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF8DC14000", "buildId" : "CC9E3394BE16397F926BE579B60EE429" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF90E1A000", "buildId" : "2F71CAF86524329EAC56C506658B4C0C" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF8E7CD000", "buildId" : "BF332AD9E89F387E92A46E1AB74BD4D9" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF93464000", "buildId" : "4BE2E642A14F340AB4825BD2AEFD9C24" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8DB3F000", "buildId" : "6763BC8E18B83AD98FFAB43713A7264F" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8EB08000", "buildId" : "D92DCBC3541C37BDAADEACC75A0C59C8" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF8FB79000", "buildId" : "4C9EB006FE1F3F8F8074DFD94CF2CE7B" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF86A7B000", "buildId" : "616FC901151E38BFB2C424A351C5FAAD" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF887CE000", "buildId" : "4FFCA2427F04365F87A6D4EFB89503C1" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF8F915000", "buildId" : "5A961E2ACFB8362BBC43122704AEB047" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF88805000", "buildId" : "B434BE5C25AB3EBDBAA75304B34E3441" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF92ED8000", "buildId" : "0D99F24E56FE380FB81B4A4C630EE587" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF8EB0A000", "buildId" : "C49275CC835A3207AFBA8C01374927B6" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF8FC47000", "buildId" : "6838A6FD86263356BB4FBB4787216207" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF9386B000", "buildId" : "92805328CD3634FF9436571AB0485072" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF9384E000", "buildId" : "21703D362DAB3D8B8442EAAB23C060D3" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF939B8000", "buildId" : "BBE558BD5E5535E489ED1AA6B056D05A" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8DB47000", "buildId" : "CE0C29A3C420339BADAA52F4683233CC" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF910B4000", "buildId" : "90D31928F48D3E37874F220A51FD9E37" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8C737000", "buildId" : "73CDC48216E33FC79BB4FBA2DA44DBC2" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8EA05000", "buildId" : "D86169F39F31377A9AF3DB17142052E4" } ] }}
       mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x1034cc839]
       mongod(_ZN5mongo12_GLOBAL__N_110abruptQuitEi+0xB6) [0x1034cc276]
       libsystem_c.dylib(_sigtramp+0x1A) [0x7fff8fb7b94a]
       ??? [0x5f]
       libsystem_c.dylib(abort+0x8F) [0x7fff8fbd2dce]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x24E) [0x10346ebfe]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEES9_b+0x775) [0x1032fe5c5]
       mongod(_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsE+0x3A) [0x1032fda9a]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE+0x77) [0x10302c307]
       mongod(_ZN5mongoL14_initAndListenEi+0xC95) [0x102d9f075]
       mongod(_ZN5mongo13initAndListenEi+0x13) [0x102d9de93]
       mongod(main+0x3BC) [0x102da2d1c]
       mongod(start+0x34) [0x102d9de74]
       ??? [0x4]
      -----  END BACKTRACE  -----
      

      There is also a 5 sec delay in wired tiger_open in this case.

            Assignee:
            Unassigned Unassigned
            Reporter:
            alex.komyagin@mongodb.com Alexander Komyagin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: