Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44589

mongodb fails to start after outage

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.20
    • Component/s: None
    • Environment:
      Debian 9 OS

       

      dccls2rh:~ # /etc/init.d/mongod start
       [....] Starting database: mongod2019-11-07T12:01:31.736-0500 I CONTROL [main] ***** SERVER RESTARTED *****
       2019-11-07T12:01:31.743-0500 I CONTROL [initandlisten] MongoDB starting : pid=4763 port=27017 dbpath=/var/lib/mongodb 64-bit host=dwdccls2rh
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] db version v3.2.20
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] allocator: tcmalloc
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] modules: none
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] build environment:
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] distmod: debian71
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] distarch: x86_64
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] target_arch: x86_64
       2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
      
      { bindIp: "10.32.13.226", port: 27017 }
      
      , security: \{ authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal:
      
      { enabled: true }
      
      }, systemLog: \{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
       2019-11-07T12:01:31.766-0500 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
       2019-11-07T12:01:31.767-0500 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
       2019-11-07T12:01:31.767-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
       2019-11-07T12:01:31.767-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
       2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (0) [1573146091:789354][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: read checksum error for 4096B block at offset 12288: block header checksum of 1043026774 doesn't match expected checksum of 1182754036
       2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (0) [1573146091:789416][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: WiredTiger.wt: encountered an illegal file format or internal value
       2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (-31804) [1573146091:789436][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
       2019-11-07T12:01:31.789-0500 I - [initandlisten] Fatal Assertion 28558
       2019-11-07T12:01:31.789-0500 I - [initandlisten]
        
       ***aborting after fassert() failure
        
        
       2019-11-07T12:01:31.807-0500 F - [initandlisten] Got signal: 6 (Aborted).
        
       0x133d2f2 0x133c219 0x133ca22 0x7f33c5b010a0 0x7f33c5797125 0x7f33c579a3a0 0x12b9732 0x109a833 0x1ac2338 0x1ac2535 0x1ac2703 0x19e50dc 0x19e5643 0x19e1ed9 0x19e68d9 0x19ffc05 0x1a3a68b 0x1ac0c0d 0x1ac1129 0x1ac134b 0x1a47e76 0x1abda9d 0x1a81e0f 0x1a81f0e 0x1a3631a 0x1081116 0x107d1f3 0xfa0cae 0x955710 0x958c0d 0x7f33c5783ead 0x9510c9
       ----- BEGIN BACKTRACE -----
      
      {"backtrace":[\\{"b":"400000","o":"F3D2F2","s":"_ZN5mongo15printStackTraceERSo"}
      
      ,\{"b":"400000","o":"F3C219"},\{"b":"400000","o":"F3CA22"},\{"b":"7F33C5AF2000","o":"F0A0"},\{"b":"7F33C5765000","o":"32125","s":"gsignal"},\{"b":"7F33C5765000","o":"353A0","s":"abort"},{"b":"400000","o":"EB9732","s":"_ZN5mongo13fassertFailedEi"},\{"b":"400000","o":"C9A833"},{"b":"400000","o":"16C2338","s":"__wt_eventv"},{"b":"400000","o":"16C2535","s":"__wt_err"},{"b":"400000","o":"16C2703","s":"__wt_panic"},{"b":"400000","o":"15E50DC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15E5643","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15E1ED9","s":"__wt_block_checkpoint_load"},\{"b":"400000","o":"15E68D9"},{"b":"400000","o":"15FFC05","s":"__wt_btree_open"},{"b":"400000","o":"163A68B","s":"__wt_conn_btree_open"},{"b":"400000","o":"16C0C0D","s":"__wt_session_get_btree"},{"b":"400000","o":"16C1129","s":"__wt_session_get_btree"},{"b":"400000","o":"16C134B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1647E76","s":"__wt_curfile_open"},\{"b":"400000","o":"16BDA9D"},{"b":"400000","o":"1681E0F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1681F0E","s":"__wt_metadata_cursor"},\{"b":"400000","o":"163631A","s":"wiredtiger_open"},\{"b":"400000","o":"C81116","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},\{"b":"400000","o":"C7D1F3"},\{"b":"400000","o":"BA0CAE","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},\{"b":"400000","o":"555710"},\{"b":"400000","o":"558C0D","s":"main"},{"b":"7F33C5765000","o":"1EEAD","s":"__libc_start_main"},\{"b":"400000","o":"5510C9"}],"processInfo":{ "mongodbVersion" : "3.2.20", "gitVersion" : "a7a144f40b70bfe290906eb33ff2714933544af8", "compiledModules" : [], "uname" :
      
      { "sysname" : "Linux", "release" : "3.2.0-6-amd64", "version" : "#1 SMP Debian 3.2.102-1", "machine" : "x86_64" }
      
      , "somap" : [ \{ "elfType" : 2, "b" : "400000", "buildId" : "D879C8CFBD84A8F158AA3605759FB36ABDEB5366" }, \{ "b" : "7FFFC06FE000", "elfType" : 3, "buildId" : "7385007A08B4A88E926289F83C79E9A4D4561B82" }, \{ "b" : "7F33C69AC000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "30890712A16EDD0CDEDC66CF8E41E7E8010C4185" }, \{ "b" : "7F33C65B2000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3579FDA4D1E7D6281305E12E320EBBBDE1306D1A" }, \{ "b" : "7F33C63AA000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CEA0BF1B6D309A394069A065AE4AE0813E5899F8" }, \{ "b" : "7F33C61A6000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "41E934315B95097DCEE271A02A0F390D07EF64AD" }, \{ "b" : "7F33C5F24000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "B62C3FA2DA9E617FF960A44873D761742BB17E90" }, \{ "b" : "7F33C5D0E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, \{ "b" : "7F33C5AF2000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C86440A1C3888E3F3EA9ADDA2F5AC4CF2512DEFC" }, \{ "b" : "7F33C5765000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8E96AC03BF6C01584DDC9750C326041E6218F0B8" }, \{ "b" : "7F33C6C0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "91C1780EFA29E8E10441CEF90A00F1CE2D67AB2A" }, \{ "b" : "7F33C554E000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x133d2f2]
       mongod(+0xF3C219) [0x133c219]
       mongod(+0xF3CA22) [0x133ca22]
       libpthread.so.0(+0xF0A0) [0x7f33c5b010a0]
       libc.so.6(gsignal+0x35) [0x7f33c5797125]
       libc.so.6(abort+0x180) [0x7f33c579a3a0]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12b9732]
       mongod(+0xC9A833) [0x109a833]
       mongod(__wt_eventv+0x440) [0x1ac2338]
       mongod(__wt_err+0x8D) [0x1ac2535]
       mongod(__wt_panic+0x2E) [0x1ac2703]
       mongod(__wt_block_extlist_read+0x6C) [0x19e50dc]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19e5643]
       mongod(__wt_block_checkpoint_load+0x359) [0x19e1ed9]
       mongod(+0x15E68D9) [0x19e68d9]
       mongod(__wt_btree_open+0xBD5) [0x19ffc05]
       mongod(__wt_conn_btree_open+0x16B) [0x1a3a68b]
       mongod(__wt_session_get_btree+0xDD) [0x1ac0c0d]
       mongod(__wt_session_get_btree+0x5F9) [0x1ac1129]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1ac134b]
       mongod(__wt_curfile_open+0x86) [0x1a47e76]
       mongod(+0x16BDA9D) [0x1abda9d]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a81e0f]
       mongod(__wt_metadata_cursor+0x7E) [0x1a81f0e]
       mongod(wiredtiger_open+0x191A) [0x1a3631a]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x5B6) [0x1081116]
       mongod(+0xC7D1F3) [0x107d1f3]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x59E) [0xfa0cae]
       mongod(+0x555710) [0x955710]
       mongod(main+0x15D) [0x958c0d]
       libc.so.6(__libc_start_main+0xFD) [0x7f33c5783ead]
       mongod(+0x5510C9) [0x9510c9]
       ----- END BACKTRACE -----
      

      failed!

        1. repair_attempt.tar.gz
          2 kB
          Carl Champain
        2. WiredTiger.turtle
          1.0 kB
          Reg Dancy
        3. WiredTiger.wt
          44 kB
          Reg Dancy

            Assignee:
            carl.champain@mongodb.com Carl Champain (Inactive)
            Reporter:
            reginald.dancy@comodo.com Reg Dancy
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: