Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19988

mongod start failed

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical - P2 Critical - P2
    • None
    • 3.0.4
    • WiredTiger
    • None
    • ALL

    Description

      I'm running 3.0.4 and have ran into a problem that appears similar to SERVER-18448. I don't have a replica. Can you help me recover the data?
      There are also some other issues that appear to have the same problem. Based on these I've already tried the following -

      • The options for repair, hazard_max=20000, etc.
      • Switching to 3.0.5 and 3.1.5.

      But all of these run into the same messages below:

      2015-08-17T19:52:44.816+0000 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2015-08-17T19:52:44.817+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-08-17T19:52:44.817+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=62G,session_max=20000,eviction=(threads_max=4),statistics=(fast),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-08-17T19:52:44.831+0000 E STORAGE  [initandlisten] WiredTiger (0) [1439841164:831079][1:0x7fc06c115b60], file:WiredTiger.wt, connection: read checksum error [4096B @ 36864, 71178466 != 2337379131]
      2015-08-17T19:52:44.831+0000 E STORAGE  [initandlisten] WiredTiger (0) [1439841164:831130][1:0x7fc06c115b60], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2015-08-17T19:52:44.831+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1439841164:831151][1:0x7fc06c115b60], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2015-08-17T19:52:44.831+0000 I -        [initandlisten] Fatal Assertion 28558
      2015-08-17T19:52:44.843+0000 I CONTROL  [initandlisten]
       0xf7a0f9 0xf19121 0xefdbb1 0xda3d5a 0x13ab769 0x13ab925 0x13abdc4 0x1300612 0x1319c0c 0x13179f8 0x1318852 0x134198b 0x13aabe0 0x1378b1b 0x133f077 0xd8e6f0 0xd8c648 0xa9baed 0x825fb2 0x7f2829 0x7fc06a774ead 0x823da9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B7A0F9"},{"b":"400000","o":"B19121"},{"b":"400000","o":"AFDBB1"},{"b":"400000","o":"9A3D5A"},{"b":"400000","o":"FAB769"},{"b":"400000","o":"FAB925"},{"b":"400000","o":"FABDC4"},{"b":"400000","o":"F00612"},{"b":"400000","o":"F19C0C"},{"b":"400000","o":"F179F8"},{"b":"400000","o":"F18852"},{"b":"400000","o":"F4198B"},{"b":"400000","o":"FAABE0"},{"b":"400000","o":"F78B1B"},{"b":"400000","o":"F3F077"},{"b":"400000","o":"98E6F0"},{"b":"400000","o":"98C648"},{"b":"400000","o":"69BAED"},{"b":"400000","o":"425FB2"},{"b":"400000","o":"3F2829"},{"b":"7FC06A756000","o":"1EEAD"},{"b":"400000","o":"423DA9"}],"processInfo":{ "mongodbVersion" : "3.0.4", "gitVersion" : "0481c958daeb2969800511e7475dc66986fa9ed5", "uname" : { "sysname" : "Linux", "release" : "3.19.0-21-generic", "version" : "#21~14.04.1-Ubuntu SMP Sun Jun 14 18:45:42 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EB7A0F7D9F202DB3ADD7637D20509D9DE82AC5A1" }, { "b" : "7FFF95FDE000", "elfType" : 3, "buildId" : "C34A51FA2CBA97D222F6B9E62BBC3114036D0A8A" }, { "b" : "7FC06BCE5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "FEF281218797AD6AE726DD5FCEDECADD9E9F51DC" }, { "b" : "7FC06BA84000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "ED2B7FC51D2E7ADD0D4F1A3667E2B6ED7257663F" }, { "b" : "7FC06B68C000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "7C7F328E0814F339D251A8F8C9459E50978EC254" }, { "b" : "7FC06B484000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F58D5DE3E7A2989E915422BA4203FE53DBA449A0" }, { "b" : "7FC06B280000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "5D1CA3A3D93ED5B6C6462FFA03E787FDBE4013A3" }, { "b" : "7FC06AF79000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "8711429397A5AF8B6269B867D830EDF6E0225B8D" }, { "b" : "7FC06ACF7000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "7F58D6664571941C86B2D969701A572AD4D7BF1D" }, { "b" : "7FC06AAE1000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7FC06A756000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "A745EBA2C16BA80AE1EF1A7A7B70740C2CF1B363" }, { "b" : "7FC06BF01000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9B23F2A44CC8CA6175CBD8D64584B1C7EA5FD18C" }, { "b" : "7FC06A53F000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf7a0f9]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf19121]
       mongod(_ZN5mongo13fassertFailedEi+0x61) [0xefdbb1]
       mongod(+0x9A3D5A) [0xda3d5a]
       mongod(__wt_eventv+0x489) [0x13ab769]
       mongod(__wt_err+0x95) [0x13ab925]
       mongod(__wt_panic+0x24) [0x13abdc4]
       mongod(__wt_bm_read+0x72) [0x1300612]
       mongod(__wt_bt_read+0x1AC) [0x1319c0c]
       mongod(__wt_btree_tree_open+0x58) [0x13179f8]
       mongod(__wt_btree_open+0xD02) [0x1318852]
       mongod(__wt_conn_btree_get+0x19B) [0x134198b]
       mongod(__wt_session_get_btree+0x500) [0x13aabe0]
       mongod(__wt_metadata_open+0x2B) [0x1378b1b]
       mongod(wiredtiger_open+0xCD7) [0x133f077]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x300) [0xd8e6f0]
       mongod(+0x98C648) [0xd8c648]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa9baed]
       mongod(_ZN5mongo13initAndListenEi+0x422) [0x825fb2]
       mongod(main+0x139) [0x7f2829]
       libc.so.6(__libc_start_main+0xFD) [0x7fc06a774ead]
       mongod(+0x423DA9) [0x823da9]
      -----  END BACKTRACE  -----
      2015-08-17T19:52:44.844+0000 I -        [initandlisten]
       
      ***aborting after fassert() failure
      

      Attachments

        1. newfiles.tgz
          264 kB
        2. WiredTiger.turtle
          0.9 kB
        3. WiredTiger.wt
          3.42 MB

        Activity

          People

            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            ankanm Ankan Mukherjee
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: