[SERVER-27538] WiredTiger.wt corrupt - unable to repair to start mongo Created: 29/Dec/16  Updated: 03/Jan/17  Resolved: 03/Jan/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Cory Robinson Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 14.04


Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Participants:

 Description   

My DB is corrupted and unusable after an intensive memory and CPU process ran for ~20 minutes. I've uploaded my WiredTiger.turle & WiredTiger.wt files below... any help is appreciated!

admin@jenkins:/data/db$ sudo mongod
2016-12-28T21:16:40.638-0800 I CONTROL  [initandlisten] MongoDB starting : pid=16897 port=27017 dbpath=/data/db 64-bit host=jenkins
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] db version v3.2.8
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] modules: none
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] build environment:
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     distmod: ubuntu1404
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] options: {}
2016-12-28T21:16:40.668-0800 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-12-28T21:16:40.668-0800 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2016-12-28T21:16:40.668-0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2016-12-28T21:16:40.668-0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (0) [1482988600:686354][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 57344: block header checksum of 3078277486 doesn't match expected checksum of 3500840698
2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (0) [1482988600:686594][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1482988600:686705][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-12-28T21:16:40.686-0800 I -        [initandlisten] Fatal Assertion 28558
2016-12-28T21:16:40.686-0800 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2016-12-28T21:16:40.709-0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x131ce72 0x131bfc9 0x131c7d2 0x7f7ca287d340 0x7f7ca24decc9 0x7f7ca24e20d8 0x12a6772 0x10a0923 0x1a7e31c 0x1a7e7dd 0x1a7ebc4 0x19affac 0x19b0523 0x19ad4b5 0x19b1569 0x19cdffc 0x1a04270 0x1a7d047 0x1a7d589 0x1a7d6ab 0x1a12858 0x1a7a1d5 0x1a44ccf 0x1a44dce 0x1a00e11 0x10889cf 0x1084c93 0xfadc48 0x9b48fd 0x96e04d 0x7f7ca24c9ec5 0x9b1037
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1BFC9"},{"b":"400000","o":"F1C7D2"},{"b":"7F7CA286D000","o":"10340"},{"b":"7F7CA24A8000","o":"36CC9","s":"gsignal"},{"b":"7F7CA24A8000","o":"3A0D8","s":"abort"},{"b":"400000","o":"EA6772","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0923"},{"b":"400000","o":"167E31C","s":"__wt_eventv"},{"b":"400000","o":"167E7DD","s":"__wt_err"},{"b":"400000","o":"167EBC4","s":"__wt_panic"},{"b":"400000","o":"15AFFAC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15B0523","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15AD4B5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15B1569"},{"b":"400000","o":"15CDFFC","s":"__wt_btree_open"},{"b":"400000","o":"1604270","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D047","s":"__wt_session_get_btree"},{"b":"400000","o":"167D589","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6AB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1612858","s":"__wt_curfile_open"},{"b":"400000","o":"167A1D5"},{"b":"400000","o":"1644CCF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1644DCE","s":"__wt_metadata_cursor"},{"b":"400000","o":"1600E11","s":"wiredtiger_open"},{"b":"400000","o":"C889CF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84C93"},{"b":"400000","o":"BADC48","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B48FD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7F7CA24A8000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-71-generic", "version" : "#114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFFF2EEF000", "elfType" : 3, "buildId" : "C2B4C020D20EE489936A4B11AF2C695FD0ECD484" }, { "b" : "7F7CA378E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F7CA33B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F7CA31AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F7CA2FA7000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F7CA2CA1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F7CA2A8B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F7CA286D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7CA24A8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F7CA39ED000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72]
 mongod(+0xF1BFC9) [0x131bfc9]
 mongod(+0xF1C7D2) [0x131c7d2]
 libpthread.so.0(+0x10340) [0x7f7ca287d340]
 libc.so.6(gsignal+0x39) [0x7f7ca24decc9]
 libc.so.6(abort+0x148) [0x7f7ca24e20d8]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a6772]
 mongod(+0xCA0923) [0x10a0923]
 mongod(__wt_eventv+0x42C) [0x1a7e31c]
 mongod(__wt_err+0x8D) [0x1a7e7dd]
 mongod(__wt_panic+0x24) [0x1a7ebc4]
 mongod(__wt_block_extlist_read+0x6C) [0x19affac]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19b0523]
 mongod(__wt_block_checkpoint_load+0x3C5) [0x19ad4b5]
 mongod(+0x15B1569) [0x19b1569]
 mongod(__wt_btree_open+0xC7C) [0x19cdffc]
 mongod(__wt_conn_btree_open+0x140) [0x1a04270]
 mongod(__wt_session_get_btree+0xE7) [0x1a7d047]
 mongod(__wt_session_get_btree+0x629) [0x1a7d589]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6ab]
 mongod(__wt_curfile_open+0x218) [0x1a12858]
 mongod(+0x167A1D5) [0x1a7a1d5]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a44ccf]
 mongod(__wt_metadata_cursor+0x7E) [0x1a44dce]
 mongod(wiredtiger_open+0x1541) [0x1a00e11]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x10889cf]
 mongod(+0xC84C93) [0x1084c93]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfadc48]
 mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9b48fd]
 mongod(main+0x15D) [0x96e04d]
 libc.so.6(__libc_start_main+0xF5) [0x7f7ca24c9ec5]
 mongod(+0x5B1037) [0x9b1037]
-----  END BACKTRACE  -----



 Comments   
Comment by Cory Robinson [ 03/Jan/17 ]

Awesome! Thank you. I've also moved my DB to a new server with more memory - hopefully no more of these issues!

Comment by Kelsey Schubert [ 03/Jan/17 ]

Hi coryrobinson42@gmail.com,

I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Thank you,
Thomas

Generated at Thu Feb 08 04:15:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.