[SERVER-25250] WiredTiger library panic Created: 25/Jul/16  Updated: 14/Aug/18  Resolved: 29/Jul/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: zhou heyue [X] Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger (1).wt     File WiredTiger.turtle     File repair_attempt.tar    
Operating System: Linux
Participants:

 Description   

My mongo was unexpectedly closed.And now it cannot work. I've tried to repair. But it doesn't work. And the log information as follow:

2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] MongoDB starting : pid=10992 port=27017 dbpath=/ick/data2 64-bit host=localhost.localdomain
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] db version v3.2.3
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] modules: none
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] build environment:
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distmod: rhel62
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     distarch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/ick/data2" } }
2016-07-25T17:29:53.847+0800 I -        [initandlisten] Detected data files in /ick/data2 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898124][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 196608: block header checksum of 675114611 doesn't match expected checksum of 1687733155
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898190][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1469438993:898209][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-07-25T17:29:53.898+0800 I -        [initandlisten] Fatal Assertion 28558
2016-07-25T17:29:53.898+0800 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2016-07-25T17:29:53.946+0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x130e1e2 0x130d119 0x130d922 0x3ee060f7e0 0x3ee0232625 0x3ee0233e05 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x199067c 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a20b7f 0x1a20c5e 0x19df078 0x107322e 0x106f6a3 0xf99268 0x974701 0x97815d 0x3ee021ed5d 0x972279
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"3EE0600000","o":"F7E0"},{"b":"3EE0200000","o":"32625","s":"gsignal"},{"b":"3EE0200000","o":"33E05","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":"400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"159067C","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"15E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"1620B7F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1620C5E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15DF078","s":"wiredtiger_open"},{"b":"400000","o":"C7322E","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6F6A3"},{"b":"400000","o":"B99268","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"574701","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57815D","s":"main"},{"b":"3EE0200000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"572279"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, { "b" : "7FFEC8C76000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5D42D8781F5FF4C06B4F839B85B21786AA44488D" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
 mongod(+0xF0D119) [0x130d119]
 mongod(+0xF0D922) [0x130d922]
 libpthread.so.0(+0xF7E0) [0x3ee060f7e0]
 libc.so.6(gsignal+0x35) [0x3ee0232625]
 libc.so.6(abort+0x175) [0x3ee0233e05]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
 mongod(+0xC8B3E3) [0x108b3e3]
 mongod(__wt_eventv+0x40C) [0x1a57e3c]
 mongod(__wt_err+0x8D) [0x1a57fdd]
 mongod(__wt_panic+0x24) [0x1a583c4]
 mongod(__wt_block_extlist_read+0x6C) [0x199067c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
 mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
 mongod(+0x15919DE) [0x19919de]
 mongod(__wt_btree_open+0xC68) [0x19ae1a8]
 mongod(__wt_conn_btree_open+0x140) [0x19e2140]
 mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
 mongod(__wt_session_get_btree+0x607) [0x1a57117]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
 mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
 mongod(+0x1654415) [0x1a54415]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a20b7f]
 mongod(__wt_metadata_cursor+0x7E) [0x1a20c5e]
 mongod(wiredtiger_open+0x15F8) [0x19df078]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x45E) [0x107322e]
 mongod(+0xC6F6A3) [0x106f6a3]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf99268]
 mongod(_ZN5mongo13initAndListenEi+0x321) [0x974701]
 mongod(main+0x15D) [0x97815d]
 libc.so.6(__libc_start_main+0xFD) [0x3ee021ed5d]
 mongod(+0x572279) [0x972279]
-----  END BACKTRACE  -----

How can I fix it?!



 Comments   
Comment by zhou heyue [X] [ 01/Aug/16 ]

Hi,
I've tried --repair.But it doesn't work.
Thanks

Comment by Kelsey Schubert [ 29/Jul/16 ]

Hi Ginger12,

Have you attempted --repair with the new files? If this does not correct the issue, it would indicate that the corruption is beyond repair.

Unfortunately, we're not able to provide further support here since the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience.

Kind regards,
Thomas

Comment by zhou heyue [X] [ 28/Jul/16 ]

Thanks very much.But there is still something wrong.The error is as follow:
2016-07-28T10:03:07.166+0800 E STORAGE [initandlisten] WiredTiger (-31803) [1469671387:
166951][2449:0x7f18582dad40], txn-recover: Recovery failed: WT_NOTFOUND: item not found

Comment by Kelsey Schubert [ 27/Jul/16 ]

Hi Ginger12,

I've attached repair_attempt.tar containing the WiredTiger.wt and WiredTiger.turtle after a repair attempt. Please extract these files in your dbpath and try starting mongod. If there is no other corruption, MongoDB should start successfully.

Kind regards,
Thomas

Comment by zhou heyue [X] [ 27/Jul/16 ]

I add the files.How can I fix my mongo?

Generated at Thu Feb 08 04:08:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.