Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43799

WiredTiger library panic

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.22
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL

      Hello!

      After backuping of my ubuntu 18 server i can't run my mongo db cause of WiredTiger error i think

       

      running:

      mongod --repair --dbpath /var/lib/mongodb/

      output:

      2019-10-03T14:25:40.981+0000 I CONTROL  [initandlisten] MongoDB starting : pid=2539 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=Dobroplat
      2019-10-03T14:25:40.981+0000 I CONTROL  [initandlisten] db version v3.2.22
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] git version: 105acca0d443f9a47c1a5bd608fd7133840a58dd
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2n  7 Dec 2017
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] modules: none
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] build environment:
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2019-10-03T14:25:40.982+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
      2019-10-03T14:25:41.004+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2019-10-03T14:25:41.004+0000 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
      2019-10-03T14:25:41.005+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2019-10-03T14:25:41.005+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2019-10-03T14:25:41.005+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
      2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (0) [1570112741:26536][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 114688: block header checksum of 1635017060 doesn't match expected checksum of 850816007
      2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (0) [1570112741:26733][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2019-10-03T14:25:41.026+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1570112741:26825][2539:0x7f71c26d6100], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2019-10-03T14:25:41.026+0000 I -        [initandlisten] Fatal Assertion 28558
      2019-10-03T14:25:41.026+0000 I -        [initandlisten]***aborting after fassert() failure
      2019-10-03T14:25:41.044+0000 F -        [initandlisten] Got signal: 6 (Aborted). 0x155c5e2 0x155b589 0x155bdf2 0x7f71c1241890 0x7f71c0e7ce97 0x7f71c0e7e801 0x14d85f3 0x12787aa 0x97e457 0x97e63d 0x97e80d 0x1ba0caf 0x1ba120b 0x1b9daad 0x1ba2367 0x1bbb6cd 0x1bf3a9b 0x1c79bcd 0x1c7a0f9 0x1c7a39c 0x1c01ae1 0x1c76a68 0x1c3bbdf 0x1c3bc9c 0x1bf012c 0x125d1ca 0x1255cb5 0x1150ebe 0x9cbb91 0x97f5ea 0x7f71c0e5fb97 0x9c6e89
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"115C5E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"115B589"},{"b":"400000","o":"115BDF2"},{"b":"7F71C122F000","o":"12890"},{"b":"7F71C0E3E000","o":"3EE97","s":"gsignal"},{"b":"7F71C0E3E000","o":"40801","s":"abort"},{"b":"400000","o":"10D85F3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E787AA"},{"b":"400000","o":"57E457","s":"__wt_eventv"},{"b":"400000","o":"57E63D","s":"__wt_err"},{"b":"400000","o":"57E80D","s":"__wt_panic"},{"b":"400000","o":"17A0CAF","s":"__wt_block_extlist_read"},{"b":"400000","o":"17A120B","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"179DAAD","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"17A2367"},{"b":"400000","o":"17BB6CD","s":"__wt_btree_open"},{"b":"400000","o":"17F3A9B","s":"__wt_conn_btree_open"},{"b":"400000","o":"1879BCD","s":"__wt_session_get_btree"},{"b":"400000","o":"187A0F9","s":"__wt_session_get_btree"},{"b":"400000","o":"187A39C","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1801AE1","s":"__wt_curfile_open"},{"b":"400000","o":"1876A68"},{"b":"400000","o":"183BBDF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"183BC9C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17F012C","s":"wiredtiger_open"},{"b":"400000","o":"E5D1CA","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E55CB5"},{"b":"400000","o":"D50EBE","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5CBB91"},{"b":"400000","o":"57F5EA","s":"main"},{"b":"7F71C0E3E000","o":"21B97","s":"__libc_start_main"},{"b":"400000","o":"5C6E89","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.22", "gitVersion" : "105acca0d443f9a47c1a5bd608fd7133840a58dd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-62-generic", "version" : "#69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C2070FF92CF0E7C7AF25D84027F691037262CEA2" }, { "b" : "7FFFCDEDE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "17ABBA294C8661C069F5408AC1D74F121AD71734" }, { "b" : "7F71C2253000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0D054641049B9747C05D030262295DFDFDD3055D" }, { "b" : "7F71C1E10000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "9C228817BA6E0730F4FCCFAC6E033BD1E0C5620A" }, { "b" : "7F71C1C08000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F71C1A04000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F71C1666000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F71C144E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F71C122F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F71C0E3E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F71C24BB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x155c5e2]
       mongod(+0x115B589) [0x155b589]
       mongod(+0x115BDF2) [0x155bdf2]
       libpthread.so.0(+0x12890) [0x7f71c1241890]
       libc.so.6(gsignal+0xC7) [0x7f71c0e7ce97]
       libc.so.6(abort+0x141) [0x7f71c0e7e801]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d85f3]
       mongod(+0xE787AA) [0x12787aa]
       mongod(__wt_eventv+0x3BA) [0x97e457]
       mongod(__wt_err+0x8B) [0x97e63d]
       mongod(__wt_panic+0x2E) [0x97e80d]
       mongod(__wt_block_extlist_read+0x6F) [0x1ba0caf]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x1ba120b]
       mongod(__wt_block_checkpoint_load+0x24D) [0x1b9daad]
       mongod(+0x17A2367) [0x1ba2367]
       mongod(__wt_btree_open+0xB0D) [0x1bbb6cd]
       mongod(__wt_conn_btree_open+0x16B) [0x1bf3a9b]
       mongod(__wt_session_get_btree+0xDD) [0x1c79bcd]
       mongod(__wt_session_get_btree+0x609) [0x1c7a0f9]
       mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c7a39c]
       mongod(__wt_curfile_open+0x151) [0x1c01ae1]
       mongod(+0x1876A68) [0x1c76a68]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1c3bbdf]
       mongod(__wt_metadata_cursor+0x3C) [0x1c3bc9c]
       mongod(wiredtiger_open+0x18DC) [0x1bf012c]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x86A) [0x125d1ca]
       mongod(+0xE55CB5) [0x1255cb5]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x1150ebe]
       mongod(+0x5CBB91) [0x9cbb91]
       mongod(main+0x73A) [0x97f5ea]
       libc.so.6(__libc_start_main+0xE7) [0x7f71c0e5fb97]
       mongod(_start+0x29) [0x9c6e89]
      -----  END BACKTRACE  -----
      Aborted (core dumped)
      

       

      Thanks!

        1. repair_attempt.tgz
          12 kB
        2. WiredTiger
          0.0 kB
        3. WiredTiger.turtle
          1.0 kB
        4. WiredTiger.wt
          220 kB
        5. WiredTigerLAS.wt
          4 kB

            Assignee:
            carl.champain@mongodb.com Carl Champain (Inactive)
            Reporter:
            dobro322@yandex.ru Vlad Kovalchuk
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: