Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32998

Mongodb Crashed with the Got signal: 6 (Aborted)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 3.2.7
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
    • Operating System:
      Linux
    • Steps To Reproduce:
      Hide

      The server is powered off and then mongo can't startup.

      Show
      The server is powered off and then mongo can't startup.

      Description

      2018-01-30T12:51:40.475+0800 I CONTROL  [main] ***** SERVER RESTARTED *****
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] MongoDB starting : pid=2621 port=27017 dbpath=/var/lib/mongodb 64-bit host=db
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] db version v3.2.7
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] modules: none
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] build environment:
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distmod: debian71
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distarch: x86_64
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, syst
      emLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
      2018-01-30T12:51:40.503+0800 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTige
      r'.
      2018-01-30T12:51:40.503+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabl
      ed=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511844][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: read checksum error for 8192B block at offset
       15413248: block header checksum of 3970207674 doesn't match expected checksum of 1545588312
      2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511873][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file fo
      rmat or internal value
      2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1517287900:511888][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PA
      NIC: WiredTiger library panic
      2018-01-30T12:51:40.511+0800 I -        [initandlisten] Fatal Assertion 28558
      2018-01-30T12:51:40.511+0800 I -        [initandlisten] 
       
      ***aborting after fassert() failure
       
       
      2018-01-30T12:51:40.533+0800 F -        [initandlisten] Got signal: 6 (Aborted).
       
       0x13043c2 0x13032e9 0x1303af2 0x7ff38993b0a0 0x7ff3895d1125 0x7ff3895d43a0 0x128a3b2 0x1072af3 0x1a7938c 0x1a7984d 0x1a79c34 0x19aa04c 0x19aa5c3 0x19a7555 0x19ab609 0x19c807c 0x19ff3d0 0x1a
      780b7 0x1a785f9 0x1a7871b 0x1a0da08 0x1a75275 0x1a3fdbf 0x1a3febe 0x19fb9c1 0x105a6c7 0x1056a43 0xf7c258 0x949b1d 0x94d71d 0x7ff3895bdead 0x946279
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F043C2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F032E9"},{"b":"400000","o":"F03AF2"},{"b":"7FF38992C000","o":"F0A0"},{"b":"7FF38959F000","o"
      :"32125","s":"gsignal"},{"b":"7FF38959F000","o":"353A0","s":"abort"},{"b":"400000","o":"E8A3B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C72AF3"},{"b":"400000","o":"167938C","s":
      "__wt_eventv"},{"b":"400000","o":"167984D","s":"__wt_err"},{"b":"400000","o":"1679C34","s":"__wt_panic"},{"b":"400000","o":"15AA04C","s":"__wt_block_extlist_read"},{"b":"400000","o":"15AA5C3
      ","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15A7555","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15AB609"},{"b":"400000","o":"15C807C","s":"__wt_btree_open"},{"b":"400000","o":"15FF3D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"16780B7","s":"__wt_session_get_btree"},{"b":"400000","o":"16785F9","s":"__wt_session_get_btree"},{"b":"400000","o":"167871B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"160DA08","s":"__wt_curfile_open"},{"b":"400000","o":"1675275"},{"b":"400000","o":"163FDBF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"163FEBE","s":"__wt_metadata_cursor"},{"b":"400000","o":"15FB9C1","s":"wiredtiger_open"},{"b":"400000","o":"C5A6C7","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C56A43"},{"b":"400000","o":"B7C258","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"549B1D","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54D71D","s":"main"},{"b":"7FF38959F000","o":"1EEAD","s":"__libc_start_main"},{"b":"400000","o":"546279"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-13-pve", "version" : "#1 SMP Mon Jul 9 08:39:20 CEST 2012", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "290161E49FF073654474160230025167BE1051AF" }, { "b" : "7FFF098FE000", "elfType" : 3, "buildId" : "1017EA1FFF5B14575F12B48EB997CEC1B048763C" }, { "b" : "7FF38A7E5000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7FF38A3EC000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "079D784F9F4A88B6D87FC3E6B0B07BFBE209540E" }, { "b" : "7FF38A1E4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "07CC333722DC4237B5F88E47A38C214FDA0B9B59" }, { "b" : "7FF389FE0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "028546D6F23D0FD259928FED6EFE53E64782EF2D" }, { "b" : "7FF389D5E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "2C5446FA41E784A1C6FDD528AB2BCC7BCA9EF06C" }, { "b" : "7FF389B48000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7FF38992C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "3F3BFFEDAF77C955862785C6CBA7670F82A05563" }, { "b" : "7FF38959F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "50B9FD83D874F18E3B385E0349FBD5D1281A19E3" }, { "b" : "7FF38AA44000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37BC6FC6FF3FA04B0A3EC20DFF215AB55EA5F5CB" }, { "b" : "7FF389388000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13043c2]
       mongod(+0xF032E9) [0x13032e9]
       mongod(+0xF03AF2) [0x1303af2]
       libpthread.so.0(+0xF0A0) [0x7ff38993b0a0]
       libc.so.6(gsignal+0x35) [0x7ff3895d1125]
       libc.so.6(abort+0x180) [0x7ff3895d43a0]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x128a3b2]
       mongod(+0xC72AF3) [0x1072af3]
       mongod(__wt_eventv+0x42C) [0x1a7938c]
       mongod(__wt_err+0x8D) [0x1a7984d]
       mongod(__wt_panic+0x24) [0x1a79c34]
       mongod(__wt_block_extlist_read+0x6C) [0x19aa04c]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19aa5c3]
       mongod(__wt_block_checkpoint_load+0x3C5) [0x19a7555]
       mongod(+0x15AB609) [0x19ab609]
       mongod(__wt_btree_open+0xC7C) [0x19c807c]
       mongod(__wt_conn_btree_open+0x140) [0x19ff3d0]
       mongod(__wt_session_get_btree+0xE7) [0x1a780b7]
       mongod(__wt_session_get_btree+0x629) [0x1a785f9]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7871b]
       mongod(__wt_curfile_open+0x218) [0x1a0da08]
       mongod(+0x1675275) [0x1a75275]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a3fdbf]
       mongod(__wt_metadata_cursor+0x7E) [0x1a3febe]
       mongod(wiredtiger_open+0x1541) [0x19fb9c1]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x567) [0x105a6c7]
       mongod(+0xC56A43) [0x1056a43]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf7c258]
       mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x949b1d]
       mongod(main+0x15D) [0x94d71d]
       libc.so.6(__libc_start_main+0xFD) [0x7ff3895bdead]
       mongod(+0x546279) [0x946279]
      -----  END BACKTRACE  -----
      

        Attachments

        1. repair-SERVER-32998.tar.gz
          16.18 MB
        2. WiredTiger.turtle
          0.9 kB
        3. WiredTiger.wt.7z
          8.90 MB

          Activity

            People

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            rory Dingfu ye
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: