Details
-
Bug
-
Resolution: Done
-
Critical - P2
-
None
-
3.0.0-rc7, 3.0.0-rc8
-
ALL
-
(copied to CRM)
Description
The power cycle testing performs random (unclean) crashes of the server. After the server reboots, mongod is run with repair.
In this particular instance, the repair failed due to file:WiredTiger.wt, connection: read checksum error:
mongod --storageEngine wiredTiger --repair
|
2015-02-02T14:58:43.936-0500 I CONTROL [initandlisten] MongoDB starting : pid=1958 port=27017 dbpath=/data/db 64-bit host=WTCrash
|
2015-02-02T14:58:43.936-0500 I CONTROL [initandlisten]
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never'
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten]
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never'
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten]
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] db version v3.0.0-rc7
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] git version: e4c60053b2967e16f765fa25d16aa6d629faa196
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] allocator: tcmalloc
|
2015-02-02T14:58:43.937-0500 I CONTROL [initandlisten] options: { repair: true, storage: { engine: "wiredTiger" } }
|
2015-02-02T14:58:43.957-0500 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
|
2015-02-02T14:58:43.957-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
|
2015-02-02T14:58:43.957-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
|
2015-02-02T14:58:43.961-0500 E STORAGE [initandlisten] WiredTiger (0) [1422907123:961875][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: read checksum error [4096B @ 28672, 3705540348 != 3545740451]
|
2015-02-02T14:58:43.962-0500 E STORAGE [initandlisten] WiredTiger (0) [1422907123:962064][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
|
2015-02-02T14:58:43.962-0500 E STORAGE [initandlisten] WiredTiger (-31804) [1422907123:962166][1958:0x7f55c79aebc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
|
2015-02-02T14:58:43.962-0500 I - [initandlisten] Fatal Assertion 28558
|
2015-02-02T14:58:43.975-0500 I CONTROL [initandlisten]
|
0xf3a9d9 0xee4b21 0xec8f41 0xd50756 0x1363350 0x1363615 0x1363a81 0x12befee 0x12bf488 0x12bc833 0x12c01c6 0x12d6849 0x12fc40b 0x13629c8 0x1330a3b 0x12f9baa 0xd52234 0xd4fd78 0xa6543d 0x7e0237 0x7e5459 0x7f55c65b8ec5 0x7ddf29
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"B3A9D9"},{"b":"400000","o":"AE4B21"},{"b":"400000","o":"AC8F41"},{"b":"400000","o":"950756"},{"b":"400000","o":"F63350"},{"b":"400000","o":"F63615"},{"b":"400000","o":"F63A81"},{"b":"400000","o":"EBEFEE"},{"b":"400000","o":"EBF488"},{"b":"400000","o":"EBC833"},{"b":"400000","o":"EC01C6"},{"b":"400000","o":"ED6849"},{"b":"400000","o":"EFC40B"},{"b":"400000","o":"F629C8"},{"b":"400000","o":"F30A3B"},{"b":"400000","o":"EF9BAA"},{"b":"400000","o":"952234"},{"b":"400000","o":"94FD78"},{"b":"400000","o":"66543D"},{"b":"400000","o":"3E0237"},{"b":"400000","o":"3E5459"},{"b":"7F55C6597000","o":"21EC5"},{"b":"400000","o":"3DDF29"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc7", "gitVersion" : "e4c60053b2967e16f765fa25d16aa6d629faa196", "uname" : { "sysname" : "Linux", "release" : "3.13.0-32-generic", "version" : "#57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFCAAFE000", "elfType" : 3 }, { "b" : "7F55C7589000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F55C7381000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F55C717D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F55C6E79000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F55C6B73000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F55C695D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F55C6597000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F55C77A7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3a9d9]
|
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee4b21]
|
mongod(_ZN5mongo13fassertFailedEi+0x61) [0xec8f41]
|
mongod(+0x950756) [0xd50756]
|
mongod(+0xF63350) [0x1363350]
|
mongod(__wt_err+0x95) [0x1363615]
|
mongod(__wt_panic+0x21) [0x1363a81]
|
mongod(__wt_block_extlist_read+0x6E) [0x12befee]
|
mongod(__wt_block_extlist_read_avail+0x28) [0x12bf488]
|
mongod(__wt_block_checkpoint_load+0x193) [0x12bc833]
|
mongod(+0xEC01C6) [0x12c01c6]
|
mongod(__wt_btree_open+0xB69) [0x12d6849]
|
mongod(__wt_conn_btree_get+0x19B) [0x12fc40b]
|
mongod(__wt_session_get_btree+0x2D8) [0x13629c8]
|
mongod(__wt_metadata_open+0x2B) [0x1330a3b]
|
mongod(wiredtiger_open+0xECA) [0x12f9baa]
|
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x304) [0xd52234]
|
mongod(+0x94FD78) [0xd4fd78]
|
mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa6543d]
|
mongod(_ZN5mongo13initAndListenEi+0x6F7) [0x7e0237]
|
mongod(main+0x139) [0x7e5459]
|
libc.so.6(__libc_start_main+0xF5) [0x7f55c65b8ec5]
|
mongod(+0x3DDF29) [0x7ddf29]
|
----- END BACKTRACE -----
|
2015-02-02T14:58:43.976-0500 I - [initandlisten]
|
|
***aborting after fassert() failure
|
Attachments
Issue Links
- is related to
-
SERVER-17587 Node crash scenario results in uncrecoverable error on subsequent startup under WiredTiger
-
- Closed
-
-
SERVER-17086 Can't start MongoDB w/ WiredTiger Engine because of Checksum Error
-
- Closed
-
-
SERVER-17210 Crash can leave data inaccessible due to empty storage.bson file
-
- Closed
-
-
SERVER-17571 WiredTiger unable to start if crash leaves 0-length WiredTiger file
-
- Closed
-
- related to
-
SERVER-17818 Abrupt termination due to service shutdown on Windows can lead to unrecoverable database under WiredTiger
-
- Closed
-
-
SERVER-17086 Can't start MongoDB w/ WiredTiger Engine because of Checksum Error
-
- Closed
-
-
SERVER-17204 Crash before completion of first checkpoint after table create can cause irrecoverable db corruption under WiredTiger
-
- Closed
-
-
SERVER-17451 WiredTiger unable to start if crash leaves 0-length journal file
-
- Closed
-
-
SERVER-17613 Unable to start mongod after unclean shutdown
-
- Closed
-