Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17152

WiredTiger file corrupted during power cycle test

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Done
    • Affects Version/s: 3.0.0-rc7, 3.0.0-rc8
    • Fix Version/s: None
    • Component/s: Storage, WiredTiger
    • Labels:
    • Operating System:
      ALL

      Description

      The power cycle testing performs random (unclean) crashes of the server. After the server reboots, mongod is run with repair.

      In this particular instance, the repair failed due to file:WiredTiger.wt, connection: read checksum error:

       mongod --storageEngine wiredTiger --repair
      2015-02-02T14:58:43.936-0500 I CONTROL  [initandlisten] MongoDB starting : pid=1958 port=27017 dbpath=/data/db 64-bit host=WTCrash
      2015-02-02T14:58:43.936-0500 I CONTROL  [initandlisten]
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten]
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten]
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] db version v3.0.0-rc7
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] git version: e4c60053b2967e16f765fa25d16aa6d629faa196
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] allocator: tcmalloc
      2015-02-02T14:58:43.937-0500 I CONTROL  [initandlisten] options: { repair: true, storage: { engine: "wiredTiger" } }
      2015-02-02T14:58:43.957-0500 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2015-02-02T14:58:43.957-0500 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-02-02T14:58:43.957-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-02-02T14:58:43.961-0500 E STORAGE  [initandlisten] WiredTiger (0) [1422907123:961875][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: read checksum error [4096B @ 28672, 3705540348 != 3545740451]
      2015-02-02T14:58:43.962-0500 E STORAGE  [initandlisten] WiredTiger (0) [1422907123:962064][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2015-02-02T14:58:43.962-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1422907123:962166][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2015-02-02T14:58:43.962-0500 I -        [initandlisten] Fatal Assertion 28558
      2015-02-02T14:58:43.975-0500 I CONTROL  [initandlisten]
       0xf3a9d9 0xee4b21 0xec8f41 0xd50756 0x1363350 0x1363615 0x1363a81 0x12befee 0x12bf488 0x12bc833 0x12c01c6 0x12d6849 0x12fc40b 0x13629c8 0x1330a3b 0x12f9baa 0xd52234 0xd4fd78 0xa6543d 0x7e0237 0x7e5459 0x7f55c65b8ec5 0x7ddf29
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B3A9D9"},{"b":"400000","o":"AE4B21"},{"b":"400000","o":"AC8F41"},{"b":"400000","o":"950756"},{"b":"400000","o":"F63350"},{"b":"400000","o":"F63615"},{"b":"400000","o":"F63A81"},{"b":"400000","o":"EBEFEE"},{"b":"400000","o":"EBF488"},{"b":"400000","o":"EBC833"},{"b":"400000","o":"EC01C6"},{"b":"400000","o":"ED6849"},{"b":"400000","o":"EFC40B"},{"b":"400000","o":"F629C8"},{"b":"400000","o":"F30A3B"},{"b":"400000","o":"EF9BAA"},{"b":"400000","o":"952234"},{"b":"400000","o":"94FD78"},{"b":"400000","o":"66543D"},{"b":"400000","o":"3E0237"},{"b":"400000","o":"3E5459"},{"b":"7F55C6597000","o":"21EC5"},{"b":"400000","o":"3DDF29"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc7", "gitVersion" : "e4c60053b2967e16f765fa25d16aa6d629faa196", "uname" : { "sysname" : "Linux", "release" : "3.13.0-32-generic", "version" : "#57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFCAAFE000", "elfType" : 3 }, { "b" : "7F55C7589000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F55C7381000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F55C717D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F55C6E79000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F55C6B73000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F55C695D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F55C6597000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F55C77A7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3a9d9]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee4b21]
       mongod(_ZN5mongo13fassertFailedEi+0x61) [0xec8f41]
       mongod(+0x950756) [0xd50756]
       mongod(+0xF63350) [0x1363350]
       mongod(__wt_err+0x95) [0x1363615]
       mongod(__wt_panic+0x21) [0x1363a81]
       mongod(__wt_block_extlist_read+0x6E) [0x12befee]
       mongod(__wt_block_extlist_read_avail+0x28) [0x12bf488]
       mongod(__wt_block_checkpoint_load+0x193) [0x12bc833]
       mongod(+0xEC01C6) [0x12c01c6]
       mongod(__wt_btree_open+0xB69) [0x12d6849]
       mongod(__wt_conn_btree_get+0x19B) [0x12fc40b]
       mongod(__wt_session_get_btree+0x2D8) [0x13629c8]
       mongod(__wt_metadata_open+0x2B) [0x1330a3b]
       mongod(wiredtiger_open+0xECA) [0x12f9baa]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x304) [0xd52234]
       mongod(+0x94FD78) [0xd4fd78]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa6543d]
       mongod(_ZN5mongo13initAndListenEi+0x6F7) [0x7e0237]
       mongod(main+0x139) [0x7e5459]
       libc.so.6(__libc_start_main+0xF5) [0x7f55c65b8ec5]
       mongod(+0x3DDF29) [0x7ddf29]
      -----  END BACKTRACE  -----
      2015-02-02T14:58:43.976-0500 I -        [initandlisten]
       
      ***aborting after fassert() failure

      1. cap-1863.js
        2 kB
        Jonathan Abrahams
      2. CAP-1863-novalidate.js
        3 kB
        Jonathan Abrahams
      3. mongod.log
        1.78 MB
        Jonathan Abrahams
      4. mongod.log-2015-02-02.tar.gz
        3.81 MB
        Jonathan Abrahams
      5. mongod-wiredTiger.log
        3.94 MB
        Jonathan Abrahams
      6. syslog-feb2.gz
        10.89 MB
        Jonathan Abrahams
      7. wt-corrupted1.tar.gz
        8.62 MB
        Jonathan Abrahams
      8. wt-failure-3.tar.gz
        5 kB
        Jonathan Abrahams

        Issue Links

          Activity

          Hide
          pasette Dan Pasette added a comment -

          We need exact reproduction steps.
          How much data is in this instance? If not large, can you make the entire contents of the dbpath available?

          Show
          pasette Dan Pasette added a comment - We need exact reproduction steps. How much data is in this instance? If not large, can you make the entire contents of the dbpath available?
          Hide
          jonathan.abrahams Jonathan Abrahams added a comment -

          Logfile of mongod which has the remove error

          Show
          jonathan.abrahams Jonathan Abrahams added a comment - Logfile of mongod which has the remove error
          Hide
          jonathan.abrahams Jonathan Abrahams added a comment -

          Power cycle client with novalidate (includes remove)

          Show
          jonathan.abrahams Jonathan Abrahams added a comment - Power cycle client with novalidate (includes remove)
          Hide
          jonathan.abrahams Jonathan Abrahams added a comment -

          Note - remove issue only observed in 3.0.1-rc0 with Ubuntu 14 VM hosted in KVM/QEMU. Not observed in CentOS 6 VM hosted in VMware.

          Show
          jonathan.abrahams Jonathan Abrahams added a comment - Note - remove issue only observed in 3.0.1-rc0 with Ubuntu 14 VM hosted in KVM/QEMU. Not observed in CentOS 6 VM hosted in VMware.
          Hide
          jonathan.abrahams Jonathan Abrahams added a comment - - edited

          Since this spawned all the linked tickets, there's no further work to be done here.

          Show
          jonathan.abrahams Jonathan Abrahams added a comment - - edited Since this spawned all the linked tickets, there's no further work to be done here.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: