[SERVER-27648] Mongo repair failing . Not able to connect to mongo Created: 12/Jan/17  Updated: 13/Aug/18  Resolved: 12/Jan/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Saurabh Dutta Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envns, rns, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.tar.gz     File WiredTiger.turtle     File WiredTiger.wt    
Operating System: Linux
Participants:

 Description   

2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] MongoDB starting : pid=31896 port=27017 dbpath=/var/lib/mongodb 64-bit host=vps306768
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] db version v3.2.9
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips  1 Mar 2016
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] modules: none
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] build environment:
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     distarch: x86_64
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger" } }
2017-01-12T06:21:26.966+0100 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2017-01-12T06:21:26.966+0100 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (0) [1484198486:986082][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 2750094563 doesn't match expected checksum of 1661035712
2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (0) [1484198486:986144][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (-31804) [1484198486:986158][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-01-12T06:21:26.986+0100 I -        [initandlisten] Fatal Assertion 28558
2017-01-12T06:21:26.986+0100 I -        [initandlisten]
 
***aborting after fassert() failure
 
 
2017-01-12T06:21:27.005+0100 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x1521902 0x15208a9 0x1521112 0x7f004873d3d0 0x7f0048398418 0x7f004839a01a 0x14a3713 0x12480f2 0x1c371b7 0x1c375ae 0x1c379b4 0x1b6421f 0x1b6474b 0x1b613e8 0x1b65827 0x1b83008 0x1bb77d0 0x1c35fcb 0x1c36560 0x1c366dc 0x1bc6bb8 0x1c33208 0x1bfc56f 0x1bfc62c 0x1bb3f2a 0x122cbae 0x1225f85 0x1121898 0x9c0070 0x975802 0x7f0048383830 0x9ba699
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"1121902","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11208A9"},{"b":"400000","o":"1121112"},{"b":"7F004872C000","o":"113D0"},{"b":"7F0048363000","o":"35418","s":"gsignal"},{"b":"7F0048363000","o":"3701A","s":"abort"},{"b":"400000","o":"10A3713","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E480F2"},{"b":"400000","o":"18371B7","s":"__wt_eventv"},{"b":"400000","o":"18375AE","s":"__wt_err"},{"b":"400000","o":"18379B4","s":"__wt_panic"},{"b":"400000","o":"176421F","s":"__wt_block_extlist_read"},{"b":"400000","o":"176474B","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"17613E8","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"1765827"},{"b":"400000","o":"1783008","s":"__wt_btree_open"},{"b":"400000","o":"17B77D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"1835FCB","s":"__wt_session_get_btree"},{"b":"400000","o":"1836560","s":"__wt_session_get_btree"},{"b":"400000","o":"18366DC","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17C6BB8","s":"__wt_curfile_open"},{"b":"400000","o":"1833208"},{"b":"400000","o":"17FC56F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"17FC62C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17B3F2A","s":"wiredtiger_open"},{"b":"400000","o":"E2CBAE","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E25F85"},{"b":"400000","o":"D21898","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5C0070","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"575802","s":"main"},{"b":"7F0048363000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5BA699","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-22-generic", "version" : "#40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "1F47133C759EFF79F90A6EBF640DC97B744D70C1" }, { "b" : "7FFD695E0000", "elfType" : 3, "buildId" : "0BFAB3B46E2221A329F60F59D05CBFBED1FECA7C" }, { "b" : "7F00496CF000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "822754695E4B31AE82937258BDFF3D52EFA0BA36" }, { "b" : "7F0049274000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B75A96C59BE1B5B54FBF1A91ED722BEC9406288E" }, { "b" : "7F004906C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A779DBCB3A477DC0C8D09B60FAC7335D396C19DF" }, { "b" : "7F0048E68000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "A2ADF3615338D49C702C41EB83A99AB743D2B574" }, { "b" : "7F0048B5F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "5C4078C04888A418F3DB0868702ECFDB35B3AD8B" }, { "b" : "7F0048949000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F004872C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B77847CC9CACBCA3B5753D0D25A32E5795AFE75B" }, { "b" : "7F0048363000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "369DE0E1D833CAA693AF17F17C83BA937F0A4DAD" }, { "b" : "7F0049938000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EDFA6D46E00CA97F349FDD3333D88493D442932C" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1521902]
 mongod(+0x11208A9) [0x15208a9]
 mongod(+0x1121112) [0x1521112]
 libpthread.so.0(+0x113D0) [0x7f004873d3d0]
 libc.so.6(gsignal+0x38) [0x7f0048398418]
 libc.so.6(abort+0x16A) [0x7f004839a01a]
 mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14a3713]
 mongod(+0xE480F2) [0x12480f2]
 mongod(__wt_eventv+0x4F7) [0x1c371b7]
 mongod(__wt_err+0x8E) [0x1c375ae]
 mongod(__wt_panic+0x24) [0x1c379b4]
 mongod(__wt_block_extlist_read+0x6F) [0x1b6421f]
 mongod(__wt_block_extlist_read_avail+0x2B) [0x1b6474b]
 mongod(__wt_block_checkpoint_load+0x1F8) [0x1b613e8]
 mongod(+0x1765827) [0x1b65827]
 mongod(__wt_btree_open+0xBA8) [0x1b83008]
 mongod(__wt_conn_btree_open+0x140) [0x1bb77d0]
 mongod(__wt_session_get_btree+0xEB) [0x1c35fcb]
 mongod(__wt_session_get_btree+0x680) [0x1c36560]
 mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c366dc]
 mongod(__wt_curfile_open+0x218) [0x1bc6bb8]
 mongod(+0x1833208) [0x1c33208]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1bfc56f]
 mongod(__wt_metadata_cursor+0x3C) [0x1bfc62c]
 mongod(wiredtiger_open+0x158A) [0x1bb3f2a]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xBCE) [0x122cbae]
 mongod(+0xE25F85) [0x1225f85]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1121898]
 mongod(_ZN5mongo13initAndListenEi+0x480) [0x9c0070]
 mongod(main+0x712) [0x975802]
 libc.so.6(__libc_start_main+0xF0) [0x7f0048383830]
 mongod(_start+0x29) [0x9ba699]
-----  END BACKTRACE  -----
mongorepair.sh: line 5: 31896 Aborted                 mongod --dbpath /var/lib/mongodb --repair --storageEngine wiredTiger
Mongo Repair Complete
Starting Mongo Server

I tried to revover data using
http://www.alexbevi.com/blog/2016/02/10/recovering-a-wiredtiger-collection-from-a-corrupt-mongodb-installation/

But this is also giving me error while recovery

 
root@vps306768:~/wiredtiger-2.9.1# ./wt -v -h ../backup/mongodb -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-8--2945606112955784657.wt
[1484199812:302917][32152:0x7f757c003700], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 2750094563 doesn't match expected checksum of 1661035712
[1484199812:303046][32152:0x7f757c003700], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
[1484199812:303094][32152:0x7f757c003700], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
lt-wt: WT_PANIC: WiredTiger library panic

Attached : WiredTiger.wt and WiredTiger.turtle



 Comments   
Comment by Mark Agarunov [ 12/Jan/17 ]

Hello saurabh73,

I'm glad to hear the repaired files fixed your issue. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thanks,
Mark

Comment by Saurabh Dutta [ 12/Jan/17 ]

Just need one quick suggestion.
My mongod service get's terminated by server instance repeatedly. Any suggestion how to fix that?

Comment by Saurabh Dutta [ 12/Jan/17 ]

Thanks a lot for such a quick response.
The given fix is working and I am able to restore my data with given fix.
Thanks a ton again.

Comment by Mark Agarunov [ 12/Jan/17 ]

Hello saurabh73,

I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? Please note that the method described in the linked blog post is not supported and may lead to further corruption in some circumstances.

Thanks,
Mark

Generated at Thu Feb 08 04:15:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.