Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34354

Mongodb do not start up again after a power outage

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.12
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL

      Hello, we have a problem with Mongo after a power outage. Mongo did not start the status this:
      ● mongodb.service - High-performance, schema-free document-oriented database
      Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
      Active: failed (Result: core-dump) since Fri 2018-04-06 08:02:31 UTC; 4s ago
      Process: 12806 ExecStart = / usr / bin / mongod --quiet --config /etc/mongod.conf (code = dumped, signal
      Main PID: 12806 (code = dumped, signal = ABRT)
      Apr 06 08:02:30 nondb02.mm4.com systemd [1]: Started High-performance, schema-free document-orient
      Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Main process exited, code = dumped, st
      Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Unit entered failed state.
      Apr 06 08:02:31 nondb02.mm4.com systemd [1]: mongodb.service: Failed with result 'core-dump'.

      In the logs there was an error by the WiredTiger.wt checksum, we fixed it by replacing the files from adjacent ticket.

      And now we have next trouble:
      service status exactly the same.
      the log is written below:

      2017-12-26T09:15:01.616+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] MongoDB starting : pid=21247 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] db version v3.2.12
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] modules: none
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] build environment:
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-12-26T09:15:01.624+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var
      /lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
      2017-12-26T09:15:01.646+0000 E NETWORK  [initandlisten] listen(): bind() failed errno:98 Address already in use for socket: 172.21.23.24:27017
      2017-12-26T09:15:01.646+0000 E NETWORK  [initandlisten]   addr already in use
      2017-12-26T09:15:01.646+0000 E STORAGE  [initandlisten] Failed to set up sockets during startup.
      2017-12-26T09:15:01.646+0000 I CONTROL  [initandlisten] dbexit:  rc: 48
      2018-04-06T07:32:03.493+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1435 port=27017 dbpath=/var/lib/mongodb 64-bit host=nondb02.mm4.com
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] db version v3.2.12
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] modules: none
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] build environment:
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-04-06T07:32:03.503+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "172.21.23.24,127.0.0.1", port: 27017 }, replication: { replSetName: "rs0" }, storage: { dbPath: "/var
      /lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
      2018-04-06T07:32:03.531+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2018-04-06T07:32:03.531+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
      2018-04-06T07:32:03.531+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2018-04-06T07:32:03.531+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,
      path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522999923:916176][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header c
      hecksum of 758591789 doesn't match expected checksum of 1262014029
      2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522999923:916233][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2018-04-06T07:32:03.916+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1522999923:916242][1435:0x7fdd55074c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTige
      r library panic
      2018-04-06T07:32:03.916+0000 I -        [initandlisten] Fatal Assertion 28558
      2018-04-06T07:32:03.916+0000 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2018-04-06T07:32:04.040+0000 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x1533852 0x15327f9 0x1533062 0x7fdd53c7c390 0x7fdd538d7428 0x7fdd538d902a 0x14b57f3 0x125b4d2 0x96f272 0x96f343 0x96f509 0x1b7681f 0x1b76d7b 0x1b7371d 0x1b77eb7 0x1b94e6c 0x1bca33b 0x1c52eb2 0x1c53585 0x1c53a7c 0x1bd9ac1 0x1c48d68 0x1c148df 0x1c149ac 0x1bc6816 0x123f568 0x1238735 0x1133f38 0x9bba60 0x970392 0x7fdd538c2830 0x9b6089
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"1133852","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11327F9"},{"b":"400000","o":"1133062"},{"b":"7FDD53C6B000","o":"11390"},{"b":"7FDD538A2000","o":"35428","s":"gsignal"},{"b":"7FDD538A2000","o":"3702A","s":"abort"},{"b":"400000","o":"10B57F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E5B4D2"},{"b":"400000","o":"56F272"},{"b":"400000","o":"56F343","s":"__wt_err"},{"b":"400000","o":"56F509","s":"__wt_panic"},{"b":"400000","o":"177681F"},{"b":"400000","o":"1776D7B"},{"b":"400000","o":"177371D"},{"b":"400000","o":"1777EB7"},{"b":"400000","o":"1794E6C"},{"b":"400000","o":"17CA33B"},{"b":"400000","o":"1852EB2"},{"b":"400000","o":"1853585"},{"b":"400000","o":"1853A7C"},{"b":"400000","o":"17D9AC1"},{"b":"400000","o":"1848D68"},{"b":"400000","o":"18148DF"},{"b":"400000","o":"18149AC"},{"b":"400000","o":"17C6816","s":"wiredtiger_open"},{"b":"400000","o":"E3F568","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E38735"},{"b":"400000","o":"D33F38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BBA60","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"570392","s":"main"},{"b":"7FDD538A2000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B6089","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "u
      ame" : { "sysname" : "Linux", "release" : "4.4.0-66-generic", "version" : "#87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7877295E4
      A95F29C74796963664BD1115CFE251E" }, { "b" : "7FFEDABCB000", "elfType" : 3, "buildId" : "5CCDAB3B3062DC906D26EBB5A688D41BFE5EFD75" }, { "b" : "7FDD54BF7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfT
      ype" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FDD547B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4
      F" }, { "b" : "7FDD545AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FDD543A7000", "path" : "/lib/x86_64-linux-gnu/libdl.so
      .2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FDD5409E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B14
      89" }, { "b" : "7FDD53E88000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FDD53C6B000", "path" : "/lib/x86_64-linux-gnu/libp
      thread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FDD538A2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6
      E4E68692053" }, { "b" : "7FDD54E60000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1533852]
       mongod(+0x11327F9) [0x15327f9]
       mongod(+0x1133062) [0x1533062]
       libpthread.so.0(+0x11390) [0x7fdd53c7c390]
       libc.so.6(gsignal+0x38) [0x7fdd538d7428]
       libc.so.6(abort+0x16A) [0x7fdd538d902a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14b57f3]
       mongod(+0xE5B4D2) [0x125b4d2]
       mongod(+0x56F272) [0x96f272]
       mongod(__wt_err+0x8B) [0x96f343]
       mongod(__wt_panic+0x24) [0x96f509]
       mongod(+0x177681F) [0x1b7681f]
       mongod(+0x1776D7B) [0x1b76d7b]
       mongod(+0x177371D) [0x1b7371d]
       mongod(+0x1777EB7) [0x1b77eb7]
       mongod(+0x1794E6C) [0x1b94e6c]
       mongod(+0x17CA33B) [0x1bca33b]
       mongod(+0x1852EB2) [0x1c52eb2]
       mongod(+0x1853585) [0x1c53585]
       mongod(+0x1853A7C) [0x1c53a7c]
       mongod(+0x17D9AC1) [0x1bd9ac1]
       mongod(+0x1848D68) [0x1c48d68]
       mongod(+0x18148DF) [0x1c148df]
       mongod(+0x18149AC) [0x1c149ac]
       mongod(wiredtiger_open+0x15D6) [0x1bc6816]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xDD8) [0x123f568]
       mongod(+0xE38735) [0x1238735]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1133f38]
       mongod(_ZN5mongo13initAndListenEi+0x480) [0x9bba60]
       mongod(main+0x712) [0x970392]
       libc.so.6(__libc_start_main+0xF0) [0x7fdd538c2830]
       mongod(_start+0x29) [0x9b6089]
      -----  END BACKTRACE  -----
      

      In the logs there was an error by the BT checkbox, we fixed it by replacing the files 1

        1. repair_attempt.tar.gz
          11 kB
          Kelsey Schubert
        2. WiredTiger.turtle
          1.0 kB
          Pavel [X]
        3. WiredTiger.wt
          196 kB
          Pavel [X]

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            iArcher Pavel [X]
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: