Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35033

Fatal Assertion 28558

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL

      Hi, after several attempt, I can't fix my database.

      the server crashed. here's what happens.

      root@vmi157799:~# mongod --repair --dbpath /var/lib/mongodb/
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] MongoDB starting : pid=22978 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=vmi157799.contaboserver.net
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] db version v3.2.18
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] git version: 4c1bae566c0c00f996a2feb16febf84936ecaf6f
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] modules: none
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] build environment:
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     distarch: x86_64
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-05-16T23:50:02.965+0200 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } }
      2018-05-16T23:50:02.996+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2018-05-16T23:50:02.996+0200 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2018-05-16T23:50:02.996+0200 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=28G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
      2018-05-16T23:50:03.007+0200 E STORAGE  [initandlisten] WiredTiger (0) [1526507403:7255][22978:0x7f9783a9ac80], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
      2018-05-16T23:50:03.007+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1526507403:7331][22978:0x7f9783a9ac80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2018-05-16T23:50:03.007+0200 I -        [initandlisten] Fatal Assertion 28558
      2018-05-16T23:50:03.007+0200 I -        [initandlisten]***aborting after fassert() failure
      2018-05-16T23:50:03.023+0200 F -        [initandlisten] Got signal: 6 (Aborted). 0x1559492 0x1558439 0x1558ca2 0x7f978269f390 0x7f97822f9428 0x7f97822fb02a 0x14d5563 0x1275b5a 0x97de59 0x97e03f 0x97e205 0x1c3988c 0x1c381da 0x1bef994 0x1c75add 0x1c76009 0x1c762ac 0x1bfdae1 0x1c72978 0x1c379af 0x1c37a6c 0x1bec12c 0x125a57a 0x1253065 0x114e28e 0x9cb181 0x97efda 0x7f97822e4830 0x9c6479
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"1159492","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"1158439"},{"b":"400000","o":"1158CA2"},{"b":"7F978268E000","o":"11390"},{"b":"7F97822C4000","o":"35428","s":"gsignal"},{"b":"7F97822C4000","o":"3702A","s":"abort"},{"b":"400000","o":"10D5563","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E75B5A"},{"b":"400000","o":"57DE59","s":"__wt_eventv"},{"b":"400000","o":"57E03F","s":"__wt_err"},{"b":"400000","o":"57E205","s":"__wt_panic"},{"b":"400000","o":"183988C","s":"__wt_turtle_read"},{"b":"400000","o":"18381DA","s":"__wt_metadata_search"},{"b":"400000","o":"17EF994","s":"__wt_conn_btree_open"},{"b":"400000","o":"1875ADD","s":"__wt_session_get_btree"},{"b":"400000","o":"1876009","s":"__wt_session_get_btree"},{"b":"400000","o":"18762AC","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17FDAE1","s":"__wt_curfile_open"},{"b":"400000","o":"1872978"},{"b":"400000","o":"18379AF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1837A6C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17EC12C","s":"wiredtiger_open"},{"b":"400000","o":"E5A57A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E53065"},{"b":"400000","o":"D4E28E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5CB181"},{"b":"400000","o":"57EFDA","s":"main"},{"b":"7F97822C4000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5C6479","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.18", "gitVersion" : "4c1bae566c0c00f996a2feb16febf84936ecaf6f", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-104-generic", "version" : "#127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "3C61A0DBFA07F1A1069E23BC3A1ADF2E695605CE" }, { "b" : "7FFFC5FEF000", "elfType" : 3, "buildId" : "BF4B5E36B7E2464DCF29A888C247A27A3FF2BC5C" }, { "b" : "7F978361A000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F97831D6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F9782FCE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F9782DCA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F9782AC1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F97828AB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F978268E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F97822C4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F9783883000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F9781DA1000", "path" : "/usr/lib/x86_64-linux-gnu/gconv/ISO8859-1.so", "elfType" : 3, "buildId" : "BFA0A1254EC586C3AA34328A7DDAA0A4DDAEF21D" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1559492]
       mongod(+0x1158439) [0x1558439]
       mongod(+0x1158CA2) [0x1558ca2]
       libpthread.so.0(+0x11390) [0x7f978269f390]
       libc.so.6(gsignal+0x38) [0x7f97822f9428]
       libc.so.6(abort+0x16A) [0x7f97822fb02a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14d5563]
       mongod(+0xE75B5A) [0x1275b5a]
       mongod(__wt_eventv+0x3BA) [0x97de59]
       mongod(__wt_err+0x8B) [0x97e03f]
       mongod(__wt_panic+0x24) [0x97e205]
       mongod(__wt_turtle_read+0x24C) [0x1c3988c]
       mongod(__wt_metadata_search+0x8A) [0x1c381da]
       mongod(__wt_conn_btree_open+0x64) [0x1bef994]
       mongod(__wt_session_get_btree+0xDD) [0x1c75add]
       mongod(__wt_session_get_btree+0x609) [0x1c76009]
       mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c762ac]
       mongod(__wt_curfile_open+0x151) [0x1bfdae1]
       mongod(+0x1872978) [0x1c72978]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1c379af]
       mongod(__wt_metadata_cursor+0x3C) [0x1c37a6c]
       mongod(wiredtiger_open+0x18DC) [0x1bec12c]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x86A) [0x125a57a]
       mongod(+0xE53065) [0x1253065]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x114e28e]
       mongod(+0x5CB181) [0x9cb181]
       mongod(main+0x73A) [0x97efda]
       libc.so.6(__libc_start_main+0xF0) [0x7f97822e4830]
       mongod(_start+0x29) [0x9c6479]
      -----  END BACKTRACE  -----
      Aborted
      

      Any suggestoin how i can fix it? I already know that some topics here have similar issue. What files do i need to upload to fix?

      Thank you.

        1. repair_attempt.tar.gz
          354 kB
        2. WiredTiger.turtle
          1.0 kB
        3. WiredTiger.wt
          5.15 MB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            atouimi Abdelouahed TOUIMI
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: