Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37747

the process must exit and restart: WT_PANIC: WiredTiger library panic

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0 Required
    • Component/s: Shell, WiredTiger
    • Labels:
      None
    • ALL
    • Hide

      The above description is my most complete startup log.

       

       

      For more details, please see the attachment.

      Show
      The above description is my most complete startup log.     For more details, please see the attachment.

      2018-10-25T10:17:15.030+0800 W - [initandlisten] Detected unclean shutdown - /data/mongo_data/mongodb/mongod.lock is not empty.
      2018-10-25T10:17:15.030+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      2018-10-25T10:17:15.030+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=11G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2018-10-25T10:17:15.034+0800 E STORAGE [initandlisten] WiredTiger (0) [1540433835:34734][16529:0x7f1cc3334cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 94208: block header checksum of 3534655478 doesn't match expected checksum of 351635639
      2018-10-25T10:17:15.034+0800 E STORAGE [initandlisten] WiredTiger (0) [1540433835:34762][16529:0x7f1cc3334cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2018-10-25T10:17:15.034+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1540433835:34770][16529:0x7f1cc3334cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2018-10-25T10:17:15.034+0800 I - [initandlisten] Fatal Assertion 28558
      2018-10-25T10:17:15.043+0800 I CONTROL [initandlisten]
      0xf75549 0xf12271 0xef60f1 0xd93f8a 0x13ab369 0x13ab525 0x13ab9c4 0x12fe3d2 0x131811c 0x1315ef8 0x1316d52 0x1340b4b 0x13aa50b 0x13780ab 0x133e207 0xd7e61c 0xd7c3a8 0xa7dc5d 0x7f57a2 0x7fa739 0x7f1cc186f445 0x7f3549
      ----- BEGIN BACKTRACE -----

      {"backtrace":[\{"b":"400000","o":"B75549"}

      ,{"b":"400000","o":"B12271"},{"b":"400000","o":"AF60F1"},{"b":"400000","o":"993F8A"},{"b":"400000","o":"FAB369"},{"b":"400000","o":"FAB525"},{"b":"400000","o":"FAB9C4"},{"b":"400000","o":"EFE3D2"},{"b":"400000","o":"F1811C"},{"b":"400000","o":"F15EF8"},{"b":"400000","o":"F16D52"},{"b":"400000","o":"F40B4B"},{"b":"400000","o":"FAA50B"},{"b":"400000","o":"F780AB"},{"b":"400000","o":"F3E207"},{"b":"400000","o":"97E61C"},{"b":"400000","o":"97C3A8"},{"b":"400000","o":"67DC5D"},{"b":"400000","o":"3F57A2"},{"b":"400000","o":"3FA739"},{"b":"7F1CC184D000","o":"22445"},{"b":"400000","o":"3F3549"}],"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "1ef45a23a4c5e3480ac919b28afcba3c615488f2", "uname" :

      { "sysname" : "Linux", "release" : "3.10.0-693.el7.x86_64", "version" : "#1 SMP Tue Aug 22 21:09:27 UTC 2017", "machine" : "x86_64" }

      , "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7AA372EDE22BA34234ADA10A8AF2E665681140" }, { "b" : "7FFFF8CC1000", "elfType" : 3, "buildId" : "7FB8E16CEA1B913E2703A6E4159FB468CD1E3507" }, { "b" : "7F1CC2F18000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F1CC2CA6000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "ED0AC7DEB91A242C194B3DEF27A215F41CE43116" }, { "b" : "7F1CC2845000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "BC0AE9CA0705BEC1F0C0375AAD839843BB219CB1" }, { "b" : "7F1CC263D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F1CC2439000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F1CC2132000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "9589AE0FDA6AEB1183EBA1C62A328F933E7817FD" }, { "b" : "7F1CC1E30000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F1CC1C1A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F1CC184D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F1CC3134000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F1CC1600000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DA322D74F55A0C4293085371A8D0E94B5962F5E7" }, { "b" : "7F1CC1318000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B69E63024D408E400401EEA6815317BDA38FB7C2" }, { "b" : "7F1CC1114000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7F1CC0EE1000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A48639BF901DB554479BFAD114CB354CF63D7D6E" }, { "b" : "7F1CC0CCB000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F1CC0ABD000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "6FDF5B013FD2739D304CFB9D723DCBC149EE03C9" }, { "b" : "7F1CC08B9000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1CC06A0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F1CC0479000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7F1CC0217000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
      mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf75549]
      mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf12271]
      mongod(_ZN5mongo13fassertFailedEi+0x61) [0xef60f1]
      mongod(+0x993F8A) [0xd93f8a]
      mongod(__wt_eventv+0x489) [0x13ab369]
      mongod(__wt_err+0x95) [0x13ab525]
      mongod(__wt_panic+0x24) [0x13ab9c4]
      mongod(__wt_bm_read+0x72) [0x12fe3d2]
      mongod(__wt_bt_read+0x1AC) [0x131811c]
      mongod(__wt_btree_tree_open+0x58) [0x1315ef8]
      mongod(__wt_btree_open+0xD02) [0x1316d52]
      mongod(__wt_conn_btree_get+0x19B) [0x1340b4b]
      mongod(__wt_session_get_btree+0x41B) [0x13aa50b]
      mongod(__wt_metadata_open+0x2B) [0x13780ab]
      mongod(wiredtiger_open+0xCD7) [0x133e207]
      mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x30C) [0xd7e61c]
      mongod(+0x97C3A8) [0xd7c3a8]
      mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa7dc5d]
      mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f57a2]
      mongod(main+0x139) [0x7fa739]
      libc.so.6(__libc_start_main+0xF5) [0x7f1cc186f445]
      mongod(+0x3F3549) [0x7f3549]
      ----- END BACKTRACE -----
      2018-10-25T10:17:15.043+0800 I - [initandlisten]

      ***aborting after fassert() failure

        1. 20181030.tar.gz
          990 kB
          hanyudong
        2. mongodb.out
          6 kB
          hanyudong
        3. repair_attempt.tar.gz
          30 kB
          Kelsey Schubert
        4. WiredTiger.turtle
          0.9 kB
          hexiaojie
        5. WiredTiger.turtle
          0.9 kB
          hanyudong
        6. WiredTiger.wt
          4.02 MB
          hexiaojie
        7. WiredTiger.wt
          4.02 MB
          hexiaojie
        8. WiredTiger.wt
          584 kB
          hanyudong

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            hanyd hanyudong
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: