Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19654

DB recovery after system crash - illegal file format

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.1.7
    • Component/s: Storage, WiredTiger

      The power cycle test uncovered another DB recovery failure for WiredTiger:

      2015-07-29T16:01:37.496-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
      2015-07-29T16:01:37.554-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-317-wt-100k/data/wiredTiger/mongod.lock is not empty.
      2015-07-29T16:01:37.554-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-07-29T16:01:37.554-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-07-29T16:01:37.698-0400 E STORAGE  [initandlisten] WiredTiger (0) [1438200097:698491][2631:0x7f30425b5cc0], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 8192B block at offset 69632: block header checksum of 1823627471 doesn't match expected checksum of 1553302352
      2015-07-29T16:01:37.698-0400 E STORAGE  [initandlisten] WiredTiger (0) [1438200097:698554][2631:0x7f30425b5cc0], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
      2015-07-29T16:01:37.698-0400 E STORAGE  [initandlisten] WiredTiger (-31804) [1438200097:698565][2631:0x7f30425b5cc0], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2015-07-29T16:01:37.698-0400 I -        [initandlisten] Fatal Assertion 28558
      2015-07-29T16:01:37.721-0400 I CONTROL  [initandlisten]
       0x1185112 0x1133b34 0x111f465 0xf48d43 0x184ba0c 0x184bbad 0x184bf94 0x179bcb4 0x17b639a 0x17bb178 0x17b821b 0x17d4519 0x17ab249 0x17ecdc1 0x1856cd1 0x1806547 0x185729e 0x17e3ba6 0x17dd7be 0xf33957 0xf30f5e 0xe5a8b9 0x8b8fb0 0x8bdc64 0x7f30411c6af5 0x8b7209
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"D85112"},{"b":"400000","o":"D33B34"},{"b":"400000","o":"D1F465"},{"b":"400000","o":"B48D43"},{"b":"400000","o":"144BA0C"},{"b":"400000","o":"144BBAD"},{"b":"400000","o":"144BF94"},{"b":"400000","o":"139BCB4"},{"b":"400000","o":"13B639A"},{"b":"400000","o":"13BB178"},{"b":"400000","o":"13B821B"},{"b":"400000","o":"13D4519"},{"b":"400000","o":"13AB249"},{"b":"400000","o":"13ECDC1"},{"b":"400000","o":"1456CD1"},{"b":"400000","o":"1406547"},{"b":"400000","o":"145729E"},{"b":"400000","o":"13E3BA6"},{"b":"400000","o":"13DD7BE"},{"b":"400000","o":"B33957"},{"b":"400000","o":"B30F5E"},{"b":"400000","o":"A5A8B9"},{"b":"400000","o":"4B8FB0"},{"b":"400000","o":"4BDC64"},{"b":"7F30411A5000","o":"21AF5"},{"b":"400000","o":"4B7209"}],"processInfo":{ "mongodbVersion" : "3.1.7-pre-", "gitVersion" : "a06c27a9887c9dd8fc43d9e7d8727a1840292911", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-229.4.2.el7.x86_64", "version" : "#1 SMP Wed May 13 10:06:09 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF33EFE000", "elfType" : 3 }, { "b" : "7F30421A5000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F3041FA1000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F3041C9A000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F3041998000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F3041782000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F3041566000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F30411A5000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F30423AD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1185112]
       mongod(_ZN5mongo10logContextEPKc+0x134) [0x1133b34]
       mongod(_ZN5mongo13fassertFailedEi+0x55) [0x111f465]
       mongod(+0xB48D43) [0xf48d43]
       mongod(__wt_eventv+0x40C) [0x184ba0c]
       mongod(__wt_err+0x8D) [0x184bbad]
       mongod(__wt_panic+0x24) [0x184bf94]
       mongod(__wt_bm_read+0x74) [0x179bcb4]
       mongod(__wt_bt_read+0x1DA) [0x17b639a]
       mongod(__wt_cache_read+0x98) [0x17bb178]
       mongod(__wt_page_in_func+0x50B) [0x17b821b]
       mongod(__wt_row_search+0x879) [0x17d4519]
       mongod(__wt_btcur_insert+0x399) [0x17ab249]
       mongod(+0x13ECDC1) [0x17ecdc1]
       mongod(+0x1456CD1) [0x1856cd1]
       mongod(__wt_log_scan+0x8F7) [0x1806547]
       mongod(__wt_txn_recover+0x2AE) [0x185729e]
       mongod(__wt_connection_workers+0x46) [0x17e3ba6]
       mongod(wiredtiger_open+0x14BE) [0x17dd7be]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x587) [0xf33957]
       mongod(+0xB30F5E) [0xf30f5e]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x589) [0xe5a8b9]
       mongod(_ZN5mongo13initAndListenEi+0x160) [0x8b8fb0]
       mongod(main+0x154) [0x8bdc64]
       libc.so.6(__libc_start_main+0xF5) [0x7f30411c6af5]
       mongod(+0x4B7209) [0x8b7209]
      -----  END BACKTRACE  -----
      2015-07-29T16:01:37.721-0400 I -        [initandlisten]
      
      ***aborting after fassert() failure
      

      This was using a build from latest for mongo & WT:

      • WT git hash: a2d6c93989e96fccc889a7acb1a0cd36927e686e
      • Mongo git hash: a06c27a9887c9dd8fc43d9e7d8727a1840292911

        1. powertest.sh
          35 kB
          Jonathan Abrahams

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            jonathan.abrahams Jonathan Abrahams
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: