-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.2.8
-
Component/s: WiredTiger
-
None
-
Environment:Linux 14.04
My DB is corrupted and unusable after an intensive memory and CPU process ran for ~20 minutes. I've uploaded my WiredTiger.turle & WiredTiger.wt files below... any help is appreciated!
admin@jenkins:/data/db$ sudo mongod 2016-12-28T21:16:40.638-0800 I CONTROL [initandlisten] MongoDB starting : pid=16897 port=27017 dbpath=/data/db 64-bit host=jenkins 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] db version v3.2.8 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] allocator: tcmalloc 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] modules: none 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] build environment: 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] distmod: ubuntu1404 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] distarch: x86_64 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] target_arch: x86_64 2016-12-28T21:16:40.639-0800 I CONTROL [initandlisten] options: {} 2016-12-28T21:16:40.668-0800 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-12-28T21:16:40.668-0800 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. 2016-12-28T21:16:40.668-0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2016-12-28T21:16:40.668-0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-12-28T21:16:40.686-0800 E STORAGE [initandlisten] WiredTiger (0) [1482988600:686354][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 57344: block header checksum of 3078277486 doesn't match expected checksum of 3500840698 2016-12-28T21:16:40.686-0800 E STORAGE [initandlisten] WiredTiger (0) [1482988600:686594][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2016-12-28T21:16:40.686-0800 E STORAGE [initandlisten] WiredTiger (-31804) [1482988600:686705][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-12-28T21:16:40.686-0800 I - [initandlisten] Fatal Assertion 28558 2016-12-28T21:16:40.686-0800 I - [initandlisten] ***aborting after fassert() failure 2016-12-28T21:16:40.709-0800 F - [initandlisten] Got signal: 6 (Aborted). 0x131ce72 0x131bfc9 0x131c7d2 0x7f7ca287d340 0x7f7ca24decc9 0x7f7ca24e20d8 0x12a6772 0x10a0923 0x1a7e31c 0x1a7e7dd 0x1a7ebc4 0x19affac 0x19b0523 0x19ad4b5 0x19b1569 0x19cdffc 0x1a04270 0x1a7d047 0x1a7d589 0x1a7d6ab 0x1a12858 0x1a7a1d5 0x1a44ccf 0x1a44dce 0x1a00e11 0x10889cf 0x1084c93 0xfadc48 0x9b48fd 0x96e04d 0x7f7ca24c9ec5 0x9b1037 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1BFC9"},{"b":"400000","o":"F1C7D2"},{"b":"7F7CA286D000","o":"10340"},{"b":"7F7CA24A8000","o":"36CC9","s":"gsignal"},{"b":"7F7CA24A8000","o":"3A0D8","s":"abort"},{"b":"400000","o":"EA6772","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0923"},{"b":"400000","o":"167E31C","s":"__wt_eventv"},{"b":"400000","o":"167E7DD","s":"__wt_err"},{"b":"400000","o":"167EBC4","s":"__wt_panic"},{"b":"400000","o":"15AFFAC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15B0523","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15AD4B5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15B1569"},{"b":"400000","o":"15CDFFC","s":"__wt_btree_open"},{"b":"400000","o":"1604270","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D047","s":"__wt_session_get_btree"},{"b":"400000","o":"167D589","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6AB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1612858","s":"__wt_curfile_open"},{"b":"400000","o":"167A1D5"},{"b":"400000","o":"1644CCF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1644DCE","s":"__wt_metadata_cursor"},{"b":"400000","o":"1600E11","s":"wiredtiger_open"},{"b":"400000","o":"C889CF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84C93"},{"b":"400000","o":"BADC48","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B48FD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7F7CA24A8000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-71-generic", "version" : "#114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFFF2EEF000", "elfType" : 3, "buildId" : "C2B4C020D20EE489936A4B11AF2C695FD0ECD484" }, { "b" : "7F7CA378E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F7CA33B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F7CA31AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F7CA2FA7000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F7CA2CA1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F7CA2A8B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F7CA286D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7CA24A8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F7CA39ED000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72] mongod(+0xF1BFC9) [0x131bfc9] mongod(+0xF1C7D2) [0x131c7d2] libpthread.so.0(+0x10340) [0x7f7ca287d340] libc.so.6(gsignal+0x39) [0x7f7ca24decc9] libc.so.6(abort+0x148) [0x7f7ca24e20d8] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a6772] mongod(+0xCA0923) [0x10a0923] mongod(__wt_eventv+0x42C) [0x1a7e31c] mongod(__wt_err+0x8D) [0x1a7e7dd] mongod(__wt_panic+0x24) [0x1a7ebc4] mongod(__wt_block_extlist_read+0x6C) [0x19affac] mongod(__wt_block_extlist_read_avail+0x33) [0x19b0523] mongod(__wt_block_checkpoint_load+0x3C5) [0x19ad4b5] mongod(+0x15B1569) [0x19b1569] mongod(__wt_btree_open+0xC7C) [0x19cdffc] mongod(__wt_conn_btree_open+0x140) [0x1a04270] mongod(__wt_session_get_btree+0xE7) [0x1a7d047] mongod(__wt_session_get_btree+0x629) [0x1a7d589] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6ab] mongod(__wt_curfile_open+0x218) [0x1a12858] mongod(+0x167A1D5) [0x1a7a1d5] mongod(__wt_metadata_cursor_open+0x5F) [0x1a44ccf] mongod(__wt_metadata_cursor+0x7E) [0x1a44dce] mongod(wiredtiger_open+0x1541) [0x1a00e11] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x10889cf] mongod(+0xC84C93) [0x1084c93] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfadc48] mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9b48fd] mongod(main+0x15D) [0x96e04d] libc.so.6(__libc_start_main+0xF5) [0x7f7ca24c9ec5] mongod(+0x5B1037) [0x9b1037] ----- END BACKTRACE -----