[SERVER-29605] Cannot start mongod or --repair because of WiredTiger.wt checksum error Created: 13/Jun/17  Updated: 27/Jul/18  Resolved: 16/Jun/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sneha Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envns, rns, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.basecfg     File WiredTiger.turtle     File WiredTiger.wt     Text File mongod.log     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi ,
Our production environment is down because monogodb is not starting and repair option also is failing.

The log says
Cannot start mongod or --repair because of WiredTiger.wt checksum error

2017-06-02T05:24:40.881-0700 E STORAGE  [initandlisten] WiredTiger (0) [1496406280:881822][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 192512: block header checksum of 1768714355 doesn't match expected checksum of 2158401612
2017-06-02T05:24:40.881-0700 E STORAGE  [initandlisten] WiredTiger (0) [1496406280:881959][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2017-06-02T05:24:40.882-0700 E STORAGE  [initandlisten] WiredTiger (-31804) [1496406280:882145][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-06-02T05:24:40.882-0700 I -        [initandlisten] Fatal Assertion 28558
2017-06-02T05:24:40.900-0700 I CONTROL  [initandlisten]
 0xf9cfe2 0xf3a269 0xf1e416 0xdbf331 0x13f20cc 0x13f227d 0x13f26f4 0x133d07c 0x133d5d3 0x133a1f2 0x133e31e 0x13592f4 0x138481b 0x13f123b 0x13bdce3 0x1381d89 0xda956b 0xda7140 0xaa84ba 0x8265a1 0x7f42d9 0x7fcfd738876d 0x8242ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B9CFE2"},{"b":"400000","o":"B3A269"},{"b":"400000","o":"B1E416"},{"b":"400000","o":"9BF331"},{"b":"400000","o":"FF20CC"},{"b":"400000","o":"FF227D"},{"b":"400000","o":"FF26F4"},{"b":"400000","o":"F3D07C"},{"b":"400000","o":"F3D5D3"},{"b":"400000","o":"F3A1F2"},{"b":"400000","o":"F3E31E"},{"b":"400000","o":"F592F4"},{"b":"400000","o":"F8481B"},{"b":"400000","o":"FF123B"},{"b":"400000","o":"FBDCE3"},{"b":"400000","o":"F81D89"},{"b":"400000","o":"9A956B"},{"b":"400000","o":"9A7140"},{"b":"400000","o":"6A84BA"},{"b":"400000","o":"4265A1"},{"b":"400000","o":"3F42D9"},{"b":"7FCFD7367000","o":"2176D"},{"b":"400000","o":"4242AD"}],"processInfo":{ "mongodbVersion" : "3.0.8", "gitVersion" : "83d8cc25e00e42856924d84e220fbe4a839e605d", "uname" : { "sysname" : "Linux", "release" : "3.13.0-37-generic", "version" : "#64~precise1-Ubuntu SMP Wed Sep 24 21:37:11 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "BDDA1C988F0ABFD36E0CF92E44313557B4CDCF59" }, { "b" : "7FFF238FE000", "elfType" : 3, "buildId" : "6C2D7ED95200205479A32E29DEE3CD929E125B36" }, { "b" : "7FCFD897E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CF53EC11DE0453E16A0021DECCC5AE8DFF132044" }, { "b" : "7FCFD871F000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "42220C46A8D4CC0B9293D4CDF5708D6C37E3A867" }, { "b" : "7FCFD8343000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3181CF2B60453D363A56E8A78F6F9493E86F2B77" }, { "b" : "7FCFD813B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "C9D0D71097C68F201557FA08EED37724F8A10197" }, { "b" : "7FCFD7F37000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "FD9488B6A172ED4E8FFDD32F6A8F9B52AFE3BD33" }, { "b" : "7FCFD7C37000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7FCFD793B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BA13740381D9E8D88BD16A2EF8834F9E406D8542" }, { "b" : "7FCFD7725000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7FCFD7367000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AC247F95A3044C7F6C4C1F17E14B7FF06BDB2B64" }, { "b" : "7FCFD8B9B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "81065F00F6E667ABED8F4AF6CFA87DA0D01693A9" }, { "b" : "7FCFD7150000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf9cfe2]
 mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf3a269]
 mongod(_ZN5mongo13fassertFailedEi+0x66) [0xf1e416]
 mongod(+0x9BF331) [0xdbf331]
 mongod(__wt_eventv+0x49C) [0x13f20cc]
 mongod(__wt_err+0x8D) [0x13f227d]
 mongod(__wt_panic+0x24) [0x13f26f4]
 mongod(__wt_block_extlist_read+0x6C) [0x133d07c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x133d5d3]
 mongod(__wt_block_checkpoint_load+0x202) [0x133a1f2]
 mongod(+0xF3E31E) [0x133e31e]
 mongod(__wt_btree_open+0xB54) [0x13592f4]
 mongod(__wt_conn_btree_get+0x1AB) [0x138481b]
 mongod(__wt_session_get_btree+0x3EB) [0x13f123b]
 mongod(__wt_metadata_open+0x33) [0x13bdce3]
 mongod(wiredtiger_open+0xD99) [0x1381d89]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x53B) [0xda956b]
 mongod(+0x9A7140) [0xda7140]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x32A) [0xaa84ba]
 mongod(_ZN5mongo13initAndListenEi+0x421) [0x8265a1]



 Comments   
Comment by Kelsey Schubert [ 16/Jun/17 ]

Hi sneha_k_a,

I've examined the provided log files. Unfortunately, this error indicates that the disk corruption affected more than just the WiredTiger.wt file. In this situation, I would recommend checking the integrity of your disks and then restoring from a backup.

Kind regards,
Thomas

Comment by Sneha [ 16/Jun/17 ]

Hi Thomas,
Can we get an update on this.

Regards
Sneha

Comment by Sneha [ 13/Jun/17 ]

Thanks for the reply Thomas.
On replacing the files provided by you in $dbpath, the above mentioned error is gone but I am running into another issue. Hence I am attaching the mongod.log as requested by you.

Comment by Kelsey Schubert [ 13/Jun/17 ]

Hi sneha_k_a,

Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate.

While this issue is likely related to some sort of disk failure, I would recommend upgrading to the latest version of either MongoDB 3.2 or 3.4 to take advantage of significant improvements to WiredTiger.

Kind regards,
Thomas

Comment by Sneha [ 13/Jun/17 ]

Attaching the required attachments.

Generated at Thu Feb 08 04:21:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.