[SERVER-27776] WiredTiger.wt read checksum error Created: 22/Jan/17  Updated: 13/Aug/18  Resolved: 13/Feb/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dingfu ye Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rns, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive Archive.zip     File WiredTiger.turtle     File WiredTiger.turtle     Text File mongod.log     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi,
I've a problem with mongodb. I wasn't able to start it.
When I try to start I've the following errors:

2017-01-21T03:22:07.441+0800 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "10.10.11.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", j
ournal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-01-21T03:22:07.458+0800 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to
 'wiredTiger'.
2017-01-21T03:22:07.458+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=150G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fas
t),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-01-21T03:22:07.465+0800 E STORAGE  [initandlisten] WiredTiger (0) [1484940127:465543][39398:0x7f52a1effcc0], file:WiredTiger.wt, connection: read checksum error for 4096B blo
ck at offset 3244032: block header checksum of 1936550947 doesn't match expected checksum of 1947167000
2017-01-21T03:22:07.465+0800 E STORAGE  [initandlisten] WiredTiger (0) [1484940127:465569][39398:0x7f52a1effcc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an ill
egal file format or internal value
2017-01-21T03:22:07.465+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1484940127:465576][39398:0x7f52a1effcc0], file:WiredTiger.wt, connection: the process must exit and re
start: WT_PANIC: WiredTiger library panic
2017-01-21T03:22:07.465+0800 I -        [initandlisten] Fatal Assertion 28558
2017-01-21T03:22:07.465+0800 I -        [initandlisten]

Could you please help to recover my database, it's very important.
I attached two file where I think is the problem.



 Comments   
Comment by Dingfu ye [ 30/Jan/18 ]

Hi, Another instance can't startup please check the `WiredTiger.turtle` file, Thanks.

2018-01-30T12:51:40.475+0800 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] MongoDB starting : pid=2621 port=27017 dbpath=/var/lib/mongodb 64-bit host=db
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] db version v3.2.7
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] modules: none
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] build environment:
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distmod: debian71
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     distarch: x86_64
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2018-01-30T12:51:40.482+0800 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, syst
emLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-01-30T12:51:40.503+0800 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTige
r'.
2018-01-30T12:51:40.503+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabl
ed=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511844][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: read checksum error for 8192B block at offset
 15413248: block header checksum of 3970207674 doesn't match expected checksum of 1545588312
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (0) [1517287900:511873][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file fo
rmat or internal value
2018-01-30T12:51:40.511+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1517287900:511888][2621:0x7ff38ac57c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PA
NIC: WiredTiger library panic
2018-01-30T12:51:40.511+0800 I -        [initandlisten] Fatal Assertion 28558
2018-01-30T12:51:40.511+0800 I -        [initandlisten] 
 
***aborting after fassert() failure
 
 
2018-01-30T12:51:40.533+0800 F -        [initandlisten] Got signal: 6 (Aborted).
 
 0x13043c2 0x13032e9 0x1303af2 0x7ff38993b0a0 0x7ff3895d1125 0x7ff3895d43a0 0x128a3b2 0x1072af3 0x1a7938c 0x1a7984d 0x1a79c34 0x19aa04c 0x19aa5c3 0x19a7555 0x19ab609 0x19c807c 0x19ff3d0 0x1a
780b7 0x1a785f9 0x1a7871b 0x1a0da08 0x1a75275 0x1a3fdbf 0x1a3febe 0x19fb9c1 0x105a6c7 0x1056a43 0xf7c258 0x949b1d 0x94d71d 0x7ff3895bdead 0x946279
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F043C2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F032E9"},{"b":"400000","o":"F03AF2"},{"b":"7FF38992C000","o":"F0A0"},{"b":"7FF38959F000","o"
:"32125","s":"gsignal"},{"b":"7FF38959F000","o":"353A0","s":"abort"},{"b":"400000","o":"E8A3B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C72AF3"},{"b":"400000","o":"167938C","s":
"__wt_eventv"},{"b":"400000","o":"167984D","s":"__wt_err"},{"b":"400000","o":"1679C34","s":"__wt_panic"},{"b":"400000","o":"15AA04C","s":"__wt_block_extlist_read"},{"b":"400000","o":"15AA5C3
","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15A7555","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15AB609"},{"b":"400000","o":"15C807C","s":"__wt_btree_open"},{"b":"400000","o":"15FF3D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"16780B7","s":"__wt_session_get_btree"},{"b":"400000","o":"16785F9","s":"__wt_session_get_btree"},{"b":"400000","o":"167871B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"160DA08","s":"__wt_curfile_open"},{"b":"400000","o":"1675275"},{"b":"400000","o":"163FDBF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"163FEBE","s":"__wt_metadata_cursor"},{"b":"400000","o":"15FB9C1","s":"wiredtiger_open"},{"b":"400000","o":"C5A6C7","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C56A43"},{"b":"400000","o":"B7C258","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"549B1D","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54D71D","s":"main"},{"b":"7FF38959F000","o":"1EEAD","s":"__libc_start_main"},{"b":"400000","o":"546279"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-13-pve", "version" : "#1 SMP Mon Jul 9 08:39:20 CEST 2012", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "290161E49FF073654474160230025167BE1051AF" }, { "b" : "7FFF098FE000", "elfType" : 3, "buildId" : "1017EA1FFF5B14575F12B48EB997CEC1B048763C" }, { "b" : "7FF38A7E5000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "16DA62C831D8080E3BCB9A1E7738D399EA3A89C0" }, { "b" : "7FF38A3EC000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "079D784F9F4A88B6D87FC3E6B0B07BFBE209540E" }, { "b" : "7FF38A1E4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "07CC333722DC4237B5F88E47A38C214FDA0B9B59" }, { "b" : "7FF389FE0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "028546D6F23D0FD259928FED6EFE53E64782EF2D" }, { "b" : "7FF389D5E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "2C5446FA41E784A1C6FDD528AB2BCC7BCA9EF06C" }, { "b" : "7FF389B48000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, { "b" : "7FF38992C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "3F3BFFEDAF77C955862785C6CBA7670F82A05563" }, { "b" : "7FF38959F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "50B9FD83D874F18E3B385E0349FBD5D1281A19E3" }, { "b" : "7FF38AA44000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37BC6FC6FF3FA04B0A3EC20DFF215AB55EA5F5CB" }, { "b" : "7FF389388000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13043c2]
 mongod(+0xF032E9) [0x13032e9]
 mongod(+0xF03AF2) [0x1303af2]
 libpthread.so.0(+0xF0A0) [0x7ff38993b0a0]
 libc.so.6(gsignal+0x35) [0x7ff3895d1125]
 libc.so.6(abort+0x180) [0x7ff3895d43a0]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x128a3b2]
 mongod(+0xC72AF3) [0x1072af3]
 mongod(__wt_eventv+0x42C) [0x1a7938c]
 mongod(__wt_err+0x8D) [0x1a7984d]
 mongod(__wt_panic+0x24) [0x1a79c34]
 mongod(__wt_block_extlist_read+0x6C) [0x19aa04c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19aa5c3]
 mongod(__wt_block_checkpoint_load+0x3C5) [0x19a7555]
 mongod(+0x15AB609) [0x19ab609]
 mongod(__wt_btree_open+0xC7C) [0x19c807c]
 mongod(__wt_conn_btree_open+0x140) [0x19ff3d0]
 mongod(__wt_session_get_btree+0xE7) [0x1a780b7]
 mongod(__wt_session_get_btree+0x629) [0x1a785f9]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7871b]
 mongod(__wt_curfile_open+0x218) [0x1a0da08]
 mongod(+0x1675275) [0x1a75275]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a3fdbf]
 mongod(__wt_metadata_cursor+0x7E) [0x1a3febe]
 mongod(wiredtiger_open+0x1541) [0x19fb9c1]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x567) [0x105a6c7]
 mongod(+0xC56A43) [0x1056a43]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf7c258]
 mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x949b1d]
 mongod(main+0x15D) [0x94d71d]
 libc.so.6(__libc_start_main+0xFD) [0x7ff3895bdead]
 mongod(+0x546279) [0x946279]
-----  END BACKTRACE  -----

WiredTiger.turtle

Comment by Kelsey Schubert [ 30/Jan/17 ]

Hi rory,

I'm sorry the repair attempt was unsuccessful. Would you please provide the logs after attempting mongod --repair with the new files? Unfortunately, since the recovery was unsuccessful my recommendation would be to restore from a backup.

Kind regards,
Thomas

Comment by Dingfu ye [ 24/Jan/17 ]

can't startup ..

Comment by Kelsey Schubert [ 24/Jan/17 ]

Hi rory,

Thanks for providing the turtle file. I've attached a tar with a repair attempt of the files. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue?

Kind regards,
Thomas

Comment by Dingfu ye [ 24/Jan/17 ]

Hi Thomas Schubert,

Thanks very much.

Comment by Kelsey Schubert [ 23/Jan/17 ]

Hi rory,

Would you please upload the WiredTiger.turtle file so we can attempt a repair?

Thank you,
Thomas

Generated at Thu Feb 08 04:16:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.