Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17613

Unable to start mongod after unclean shutdown

    • Fully Compatible
    • ALL

      Issue Status as of Apr 02, 2015

      ISSUE SUMMARY
      After an ungraceful termination, a mongod running WiredTiger may fail to read the journal and fail to start, showing the following messages in the logs:

      2015-03-16T14:49:24.992-0400 W -        [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty.
      2015-03-16T14:49:24.992-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      ...
      2015-03-16T14:49:25.075-0400 F -        [initandlisten] Invalid access at address: 0
      

      WORKAROUNDS
      Please upgrade to to 3.0.2.

      Alternatively, a mongod from 3.0.0-rc8 can be used to recover the data files, but this is not recommended.

      AFFECTED VERSIONS
      3.0.0-rc8+ and 3.0.1

      FIX VERSION
      The fix is included in the 3.0.2 production release.

      Original description

      As part of the Power cycle testing (see SERVER-17152), the following was discovered:

      jonathan@CAP-JA-Ubuntu14:~$ sudo mongod --dbpath /data/wiredTiger --storageEngine wiredTiger
      2015-03-16T14:49:24.992-0400 W -        [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty.
      2015-03-16T14:49:24.992-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-03-16T14:49:24.992-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-03-16T14:49:25.075-0400 F -        [initandlisten] Invalid access at address: 0
      2015-03-16T14:49:25.083-0400 F -        [initandlisten] Got signal: 11 (Segmentation fault).
      
       0xf4fd39 0xf4f3b2 0xf4f70e 0x7f4bac390340 0x13576e8 0x132ad0d 0x132b31b 0x133a2ca 0x138c8a5 0x131b3c1 0x13150dc 0xd62cab 0xd60b28 0xa6f8bd 0x7e20c0 0x7e7704 0x7f4bab3b0ec5 0x7e02c9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B4FD39"},{"b":"400000","o":"B4F3B2"},{"b":"400000","o":"B4F70E"},{"b":"7F4BAC380000","o":"10340"},{"b":"400000","o":"F576E8"},{"b":"400000","o":"F2AD0D"},{"b":"400000","o":"F2B31B"},{"b":"400000","o":"F3A2CA"},{"b":"400000","o":"F8C8A5"},{"b":"400000","o":"F1B3C1"},{"b":"400000","o":"F150DC"},{"b":"400000","o":"962CAB"},{"b":"400000","o":"960B28"},{"b":"400000","o":"66F8BD"},{"b":"400000","o":"3E20C0"},{"b":"400000","o":"3E7704"},{"b":"7F4BAB38F000","o":"21EC5"},{"b":"400000","o":"3E02C9"}],"processInfo":{ "mongodbVersion" : "3.0.1-rc0", "gitVersion" : "3f999d764cdd307f779b27e3a28230b21bd43e25", "uname" : { "sysname" : "Linux", "release" : "3.16.0-31-generic", "version" : "#41~14.04.1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF64135000", "elfType" : 3 }, { "b" : "7F4BAC380000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F4BAC178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F4BABF74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F4BABC70000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F4BAB96A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F4BAB754000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4BAB38F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F4BAC59E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4fd39]
       mongod(+0xB4F3B2) [0xf4f3b2]
       mongod(+0xB4F70E) [0xf4f70e]
       libpthread.so.0(+0x10340) [0x7f4bac390340]
       mongod(__wt_struct_unpack+0x368) [0x13576e8]
       mongod(+0xF2AD0D) [0x132ad0d]
       mongod(+0xF2B31B) [0x132b31b]
       mongod(__wt_log_needs_recovery+0xCA) [0x133a2ca]
       mongod(__wt_txn_recover+0x425) [0x138c8a5]
       mongod(__wt_connection_workers+0x61) [0x131b3c1]
       mongod(wiredtiger_open+0x118C) [0x13150dc]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd62cab]
       mongod(+0x960B28) [0xd60b28]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa6f8bd]
       mongod(_ZN5mongo13initAndListenEi+0x2F0) [0x7e20c0]
       mongod(main+0x134) [0x7e7704]
       libc.so.6(__libc_start_main+0xF5) [0x7f4bab3b0ec5]
       mongod(+0x3E02C9) [0x7e02c9]
      -----  END BACKTRACE  -----
      

        1. wt301-invalidaccess.tar.gz
          139.66 MB
          Jonathan Abrahams

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            jonathan.abrahams Jonathan Abrahams
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: