Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31813

mongod crashes repeatedly

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.2.12
    • Component/s: None
    • Labels:
    • Linux
    • Hide

      NA

      Show
      NA

      Hi Team,

      mongod is crashing continously with the below logs. This is happening repeatedly because of which mongod doesnt come up.

      This is on linux platform.

      2017-10-13T16:59:49.436+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7489 port=27017 dbpath=/var/opt/lamplighter/data 64-bit host=sojo-cfmcv-01.provocraft.com
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] db version v3.2.12
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] modules: none
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] build environment:
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] options: { config: "/var/opt/lamplighter/etc/mongod.conf", storage: { dbPath: "/var/opt/lamplighter/data", engine: "wiredTiger", wiredTiger: { engineConfig: { configString: "cache_size=1536M" } } }, systemLog: { component: { accessControl: { verbosity: 0 }, command: { verbosity: 0 } }, destination: "file", logAppend: true, logRotate: "reopen", path: "/var/opt/lamplighter/log/mongodb.log", timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } }
      2017-10-13T16:59:49.495+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=1536M
      2017-10-13T16:59:49.741+0000 E STORAGE  [initandlisten] WiredTiger (-31802) [1507913989:740996][7489:0x7f4165a00c80], file:collection-40--8734011645000920999.wt, WT_SESSION.open_cursor: unable to read root page from file:collection-40--8734011645000920999.wt: WT_ERROR: non-specific WiredTiger error
      2017-10-13T16:59:49.741+0000 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 79
      2017-10-13T16:59:49.751+0000 I CONTROL  [initandlisten]
       0x130ff72 0x12bae28 0x12a314d 0x108d304 0x108bc00 0x1087c22 0x108662d 0x10771db 0xfd66d4 0xfdc472 0x1075a58 0xf996e8 0x95a91e 0x95cec0 0x9600cd 0x7f4164930635 0x9572d9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F0FF72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EBAE28","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"EA314D","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"C8D304","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C8BC00","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C87C22","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C8662D","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C771DB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BD66D4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BDC472","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C75A58"},{"b":"400000","o":"B996E8","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"55A91E"},{"b":"400000","o":"55CEC0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5600CD","s":"main"},{"b":"7F416490F000","o":"21635","s":"__libc_start_main"},{"b":"400000","o":"5572D9"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.53sf.virtual-150", "version" : "#1 SMP Thu Aug 31 19:10:36 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFEEEFE000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7F41655E7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F41653E3000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F41650E9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F4164ED4000", "path" : "/usr/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4164CB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F416490F000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F41657F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ff72]
       mongod(_ZN5mongo10logContextEPKc+0x138) [0x12bae28]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xAD) [0x12a314d]
       mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0xE4) [0x108d304]
       mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x108bc00]
       mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x1087c22]
       mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x3ED) [0x108662d]
       mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x1AB) [0x10771db]
       mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfd66d4]
       mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x6D2) [0xfdc472]
       mongod(+0xC75A58) [0x1075a58]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf996e8]
       mongod(+0x55A91E) [0x95a91e]
       mongod(_ZN5mongo13initAndListenEi+0x10) [0x95cec0]
       mongod(main+0x15D) [0x9600cd]
       libc.so.6(__libc_start_main+0xF5) [0x7f4164930635]
       mongod(+0x5572D9) [0x9572d9]
      -----  END BACKTRACE  -----
      2017-10-13T16:59:49.751+0000 I -        [initandlisten]
      
      ***aborting after invariant() failure
      
      

      Please let us know if there is any workaround.
      Can I run "mongod —repair" here? Would it help?

      Thanks,
      Sandesh

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            sandshet Sandesh Kumar Shetty
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: