Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31813

mongod crashes repeatedly

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.2.12
    • Component/s: None
    • Linux
    • Hide

      NA

      Show
      NA
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi Team,

      mongod is crashing continously with the below logs. This is happening repeatedly because of which mongod doesnt come up.

      This is on linux platform.

      2017-10-13T16:59:49.436+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7489 port=27017 dbpath=/var/opt/lamplighter/data 64-bit host=sojo-cfmcv-01.provocraft.com
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] db version v3.2.12
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] modules: none
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] build environment:
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-10-13T16:59:49.443+0000 I CONTROL  [initandlisten] options: { config: "/var/opt/lamplighter/etc/mongod.conf", storage: { dbPath: "/var/opt/lamplighter/data", engine: "wiredTiger", wiredTiger: { engineConfig: { configString: "cache_size=1536M" } } }, systemLog: { component: { accessControl: { verbosity: 0 }, command: { verbosity: 0 } }, destination: "file", logAppend: true, logRotate: "reopen", path: "/var/opt/lamplighter/log/mongodb.log", timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } }
      2017-10-13T16:59:49.495+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=1536M
      2017-10-13T16:59:49.741+0000 E STORAGE  [initandlisten] WiredTiger (-31802) [1507913989:740996][7489:0x7f4165a00c80], file:collection-40--8734011645000920999.wt, WT_SESSION.open_cursor: unable to read root page from file:collection-40--8734011645000920999.wt: WT_ERROR: non-specific WiredTiger error
      2017-10-13T16:59:49.741+0000 I -        [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 79
      2017-10-13T16:59:49.751+0000 I CONTROL  [initandlisten]
       0x130ff72 0x12bae28 0x12a314d 0x108d304 0x108bc00 0x1087c22 0x108662d 0x10771db 0xfd66d4 0xfdc472 0x1075a58 0xf996e8 0x95a91e 0x95cec0 0x9600cd 0x7f4164930635 0x9572d9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F0FF72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EBAE28","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"EA314D","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"C8D304","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C8BC00","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C87C22","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C8662D","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C771DB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BD66D4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BDC472","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C75A58"},{"b":"400000","o":"B996E8","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"55A91E"},{"b":"400000","o":"55CEC0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5600CD","s":"main"},{"b":"7F416490F000","o":"21635","s":"__libc_start_main"},{"b":"400000","o":"5572D9"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.53sf.virtual-150", "version" : "#1 SMP Thu Aug 31 19:10:36 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFEEEFE000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7F41655E7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F41653E3000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F41650E9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F4164ED4000", "path" : "/usr/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4164CB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F416490F000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F41657F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ff72]
       mongod(_ZN5mongo10logContextEPKc+0x138) [0x12bae28]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xAD) [0x12a314d]
       mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0xE4) [0x108d304]
       mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x108bc00]
       mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x1087c22]
       mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x3ED) [0x108662d]
       mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x1AB) [0x10771db]
       mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfd66d4]
       mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x6D2) [0xfdc472]
       mongod(+0xC75A58) [0x1075a58]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf996e8]
       mongod(+0x55A91E) [0x95a91e]
       mongod(_ZN5mongo13initAndListenEi+0x10) [0x95cec0]
       mongod(main+0x15D) [0x9600cd]
       libc.so.6(__libc_start_main+0xF5) [0x7f4164930635]
       mongod(+0x5572D9) [0x9572d9]
      -----  END BACKTRACE  -----
      2017-10-13T16:59:49.751+0000 I -        [initandlisten]
      
      ***aborting after invariant() failure
      
      

      Please let us know if there is any workaround.
      Can I run "mongod —repair" here? Would it help?

      Thanks,
      Sandesh

            Assignee:
            mark.agarunov Mark Agarunov (Inactive)
            Reporter:
            sandshet Sandesh Kumar Shetty
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved:
              None
              None
              None
              None