Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27947

Failed to repair mongo after system crash

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.5
    • Component/s: Admin
    • Environment:
    • Linux

      The system was crushed, but when i start mong with --repair key , i got read checksum error:

      ~[root@iocean ~]# sudo mongod --repair --dbpath /data/mongo/
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] MongoDB starting : pid=6801 port=27017 dbpath=/data/mongo/ 64-bit host=iocean
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] db version v3.2.5
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] modules: none
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] build environment:
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     distmod: rhel62
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     distarch: x86_64
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-02-08T09:48:58.720+0700 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/data/mongo/" } }
      2017-02-08T09:48:58.738+0700 I -        [initandlisten] Detected data files in /data/mongo/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-02-08T09:48:58.738+0700 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2017-02-08T09:48:58.738+0700 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (0) [1486522138:751195][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 61440: block header checksum of 842544182 doesn't match expected checksum of 4211214560
      2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (0) [1486522138:751233][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2017-02-08T09:48:58.751+0700 E STORAGE  [initandlisten] WiredTiger (-31804) [1486522138:751249][6801:0x7fcfd7254d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-02-08T09:48:58.751+0700 I -        [initandlisten] Fatal Assertion 28558
      2017-02-08T09:48:58.751+0700 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2017-02-08T09:48:58.765+0700 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x1332c72 0x1331ba9 0x13323b2 0x7fcfd5f407e0 0x7fcfd5bcf625 0x7fcfd5bd0e05 0x12ba5e2 0x10ac1c3 0x1a9761c 0x1a977bd 0x1a97ba4 0x19cc4dc 0x19cca53 0x19c9a37 0x19cdab9 0x19ea34c 0x1a1feb0 0x1a96237 0x1a96779 0x1a9689b 0x1a2dfa8 0x1a933e5 0x1a5f8cf 0x1a5f9ce 0x1a1cef9 0x1093d0f 0x1090183 0xfb8918 0x9956d6 0x9992bd 0x7fcfd5bbbd5d 0x991fc9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F32C72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F31BA9"},{"b":"400000","o":"F323B2"},{"b":"7FCFD5F31000","o":"F7E0"},{"b":"7FCFD5B9D000","o":"32625","s":"gsignal"},{"b":"7FCFD5B9D000","o":"33E05","s":"abort"},{"b":"400000","o":"EBA5E2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAC1C3"},{"b":"400000","o":"169761C","s":"__wt_eventv"},{"b":"400000","o":"16977BD","s":"__wt_err"},{"b":"400000","o":"1697BA4","s":"__wt_panic"},{"b":"400000","o":"15CC4DC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15CCA53","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15C9A37","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15CDAB9"},{"b":"400000","o":"15EA34C","s":"__wt_btree_open"},{"b":"400000","o":"161FEB0","s":"__wt_conn_btree_open"},{"b":"400000","o":"1696237","s":"__wt_session_get_btree"},{"b":"400000","o":"1696779","s":"__wt_session_get_btree"},{"b":"400000","o":"169689B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"162DFA8","s":"__wt_curfile_open"},{"b":"400000","o":"16933E5"},{"b":"400000","o":"165F8CF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"165F9CE","s":"__wt_metadata_cursor"},{"b":"400000","o":"161CEF9","s":"wiredtiger_open"},{"b":"400000","o":"C93D0F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C90183"},{"b":"400000","o":"BB8918","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5956D6","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5992BD","s":"main"},{"b":"7FCFD5B9D000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"591FC9"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "351982A2A6B22BDE14AD7D42FE7A293C7BA951AF" }, { "b" : "7FFE35F64000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "b" : "7FCFD6DD8000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "b" : "7FCFD69F4000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "b" : "7FCFD67EC000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "b" : "7FCFD65E8000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "b" : "7FCFD6364000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "b" : "7FCFD614E000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "b" : "7FCFD5F31000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "b" : "7FCFD5B9D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "b" : "7FCFD7044000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "b" : "7FCFD5959000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "b" : "7FCFD5672000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "b" : "7FCFD546E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "b" : "7FCFD5242000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "b" : "7FCFD502C000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7FCFD4E21000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "b" : "7FCFD4C1E000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7FCFD4A04000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "b" : "7FCFD47E5000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1332c72]
       mongod(+0xF31BA9) [0x1331ba9]
       mongod(+0xF323B2) [0x13323b2]
       libpthread.so.0(+0xF7E0) [0x7fcfd5f407e0]
       libc.so.6(gsignal+0x35) [0x7fcfd5bcf625]
       libc.so.6(abort+0x175) [0x7fcfd5bd0e05]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12ba5e2]
       mongod(+0xCAC1C3) [0x10ac1c3]
       mongod(__wt_eventv+0x40C) [0x1a9761c]
       mongod(__wt_err+0x8D) [0x1a977bd]
       mongod(__wt_panic+0x24) [0x1a97ba4]
       mongod(__wt_block_extlist_read+0x6C) [0x19cc4dc]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19cca53]
       mongod(__wt_block_checkpoint_load+0x3B7) [0x19c9a37]
       mongod(+0x15CDAB9) [0x19cdab9]
       mongod(__wt_btree_open+0xC7C) [0x19ea34c]
       mongod(__wt_conn_btree_open+0x140) [0x1a1feb0]
       mongod(__wt_session_get_btree+0xE7) [0x1a96237]
       mongod(__wt_session_get_btree+0x629) [0x1a96779]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9689b]
       mongod(__wt_curfile_open+0x218) [0x1a2dfa8]
       mongod(+0x16933E5) [0x1a933e5]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a5f8cf]
       mongod(__wt_metadata_cursor+0x7E) [0x1a5f9ce]
       mongod(wiredtiger_open+0x19F9) [0x1a1cef9]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x1093d0f]
       mongod(+0xC90183) [0x1090183]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfb8918]
       mongod(_ZN5mongo13initAndListenEi+0x376) [0x9956d6]
       mongod(main+0x15D) [0x9992bd]
       libc.so.6(__libc_start_main+0xFD) [0x7fcfd5bbbd5d]
       mongod(+0x591FC9) [0x991fc9]
      -----  END BACKTRACE  -----
      

      Pls support me resolve this problem,
      Thanks & Best regards,

        1. _mdb_catalog.wt
          324 kB
          Thoang Tran
        2. mongod.log
          18 kB
          Thoang Tran
        3. mongod1.log
          8 kB
          Thoang Tran
        4. repair-27947-2.tar.gz
          545 kB
          Mark Agarunov
        5. sizeStorer.wt
          344 kB
          Thoang Tran
        6. sizeStorer.wt
          68 kB
          Thoang Tran
        7. storage.bson
          0.1 kB
          Thoang Tran
        8. WiredTiger
          0.0 kB
          Thoang Tran
        9. WiredTiger
          0.0 kB
          Thoang Tran
        10. WiredTiger.turtle
          0.9 kB
          Thoang Tran
        11. WiredTiger.turtle
          0.9 kB
          Thoang Tran
        12. WiredTiger.wt
          8.48 MB
          Thoang Tran
        13. WiredTiger.wt
          1.23 MB
          Thoang Tran
        14. WiredTigerLAS.wt
          4 kB
          Thoang Tran
        15. WT-Repaired-27947.tar.gz
          71 kB
          Mark Agarunov

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            vagabond_vn Thoang Tran
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: