[SERVER-20450] unable to start server , there is some issue with wiredtiger.wt file, unable to repair Created: 17/Sep/15  Updated: 18/Sep/15  Resolved: 18/Sep/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: naveen Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.basecfg     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tgz     File sizeStorer.wt     File storage.bson    
Operating System: ALL
Steps To Reproduce:

I tried to repair it but unable to repair

Participants:

 Description   

i am unable to start my mongodb server, there is some issue with wiredtiget.wt file

Log -

2015-09-17T14:21:38.022+0530 I CONTROL  ***** SERVER RESTARTED *****
2015-09-17T14:21:38.046+0530 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-09-17T14:21:38.052+0530 E STORAGE  [initandlisten] WiredTiger (0) [1442479898:52349][7171:0x7f5498d2fc20], file:WiredTiger.wt, connection: read checksum error [4096B @ 4096, 1482042793 != 3095125505]
2015-09-17T14:21:38.052+0530 E STORAGE  [initandlisten] WiredTiger (0) [1442479898:52384][7171:0x7f5498d2fc20], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-09-17T14:21:38.052+0530 E STORAGE  [initandlisten] WiredTiger (-31804) [1442479898:52400][7171:0x7f5498d2fc20], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-09-17T14:21:38.052+0530 I -        [initandlisten] Fatal Assertion 28558
2015-09-17T14:21:38.064+0530 I CONTROL  [initandlisten] 
 0xf6a889 0xf08321 0xeec161 0xd88caa 0x139bc59 0x139be15 0x139c2b4 0x12f2fc2 0x130b60c 0x1309418 0x130a272 0x13322bb 0x139b0d3 0x136919b 0x132f9a7 0xd72b8b 0xd70a28 0xa8104d 0x7f3e62 0x7f93c4 0x33ac81ed5d 0x7f1bbd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B6A889"},{"b":"400000","o":"B08321"},{"b":"400000","o":"AEC161"},{"b":"400000","o":"988CAA"},{"b":"400000","o":"F9BC59"},{"b":"400000","o":"F9BE15"},{"b":"400000","o":"F9C2B4"},{"b":"400000","o":"EF2FC2"},{"b":"400000","o":"F0B60C"},{"b":"400000","o":"F09418"},{"b":"400000","o":"F0A272"},{"b":"400000","o":"F322BB"},{"b":"400000","o":"F9B0D3"},{"b":"400000","o":"F6919B"},{"b":"400000","o":"F2F9A7"},{"b":"400000","o":"972B8B"},{"b":"400000","o":"970A28"},{"b":"400000","o":"68104D"},{"b":"400000","o":"3F3E62"},{"b":"400000","o":"3F93C4"},{"b":"33AC800000","o":"1ED5D"},{"b":"400000","o":"3F1BBD"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.8.1.el6.x86_64", "version" : "#1 SMP Wed Jan 28 21:11:36 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "108A63CA14A4BD5E599BAC10885DBD3A85DA5439" }, { "b" : "7FFF9A0FF000", "elfType" : 3, "buildId" : "71A96B79419FE40FF1706D59ED06D6DBE37C8E97" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AC11C021C127E84A2724F3A750461A2183B1069" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "78A423B7DD3C00F7A77027D8D764BB432B57F1E7" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "C03877A9EE01DDC572E2B0F55F64C757773CF8D6" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "C9A87F6A29ED1D3CB18F539845A45FE3A9877FF1" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "1425CB3B4C2F49C8101ED9B8F1D289053B4DFA77" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54BA6B78A9220344E77463947215E42F0EABCC62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6797403AA5F8FAD8ADFF683478B45F528CE4FB0E" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "8CE28F280150E62296240E70ECAC64E4A57AB826" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "05733977F4E41652B86070B27A0CFC2C1EA7719D" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E3FA235F3BA3F776A01A18ECA737C9890F445923" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "58B696478044E028A5970D48A4ED50E164B43B36" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf6a889]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf08321]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeec161]
 mongod(+0x988CAA) [0xd88caa]
 mongod(__wt_eventv+0x489) [0x139bc59]
 mongod(__wt_err+0x95) [0x139be15]
 mongod(__wt_panic+0x24) [0x139c2b4]
 mongod(__wt_bm_read+0x72) [0x12f2fc2]
 mongod(__wt_bt_read+0x1AC) [0x130b60c]
 mongod(__wt_btree_tree_open+0x58) [0x1309418]
 mongod(__wt_btree_open+0xD02) [0x130a272]
 mongod(__wt_conn_btree_get+0x19B) [0x13322bb]
 mongod(__wt_session_get_btree+0x343) [0x139b0d3]
 mongod(__wt_metadata_open+0x2B) [0x136919b]
 mongod(wiredtiger_open+0xCD7) [0x132f9a7]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd72b8b]
 mongod(+0x970A28) [0xd70a28]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa8104d]
 mongod(_ZN5mongo13initAndListenEi+0x422) [0x7f3e62]
 mongod(main+0x134) [0x7f93c4]
 libc.so.6(__libc_start_main+0xFD) [0x33ac81ed5d]
 mongod(+0x3F1BBD) [0x7f1bbd]
-----  END BACKTRACE  -----
2015-09-17T14:21:38.064+0530 I -        [initandlisten] 
 
***aborting after fassert() failure



 Comments   
Comment by Ramon Fernandez Marina [ 18/Sep/15 ]

Thanks for the update naven@123, and glad to hear you were able to repair your data.

The bug you run into could be triggered by an unclean shutdown, and it has been fixed in MongoDB 3.0.6, that's why I recommend you upgrade. The repair process I used is specific to this bug and it requires detailed knowledge of the WT internals, so is not something that will help if you run into other issues related to corruption.

I'm going to close this ticket. Please upgrade to MongoDB 3.0.6 at your earliest convenience to avoid corruption caused by unclean shutdowns.

Regards,
Ramón.

Comment by naveen [ 18/Sep/15 ]

thanks, I am able to repair my data . But I want to repair these files at my side, can you please help in in doing this task because it is a lengthy process to contact you and wait for your reply. If there is any process to do the things at my side, please share.

Comment by Ramon Fernandez Marina [ 17/Sep/15 ]

naven@123, please find the results of the repair attempt in the repair_attempt.tgz file. Replace the WiredTiger.wt and WiredTiger.turtle in your installation with the ones from this file and try starting the server again.

Please note that you should upgrade to MongoDB 3.0.6 right away, as some earlier versions of MongoDB 3.0 may not be able to restart after an unclean shutdown.

Comment by naveen [ 17/Sep/15 ]

uploaded, please check

Comment by Ramon Fernandez Marina [ 17/Sep/15 ]

naven@123, can you please upload the WiredTiger.turtle file as well? We can them attempt a repair.

Thanks,
Ramón.

Generated at Thu Feb 08 03:54:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.