[SERVER-35945] WiredTiger.wt Read Checksum Error Created: 03/Jul/18  Updated: 14/Aug/18  Resolved: 03/Jul/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Diego Pilger Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File WiredTiger     File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt    
Operating System: ALL
Steps To Reproduce:

2018-07-02T07:52:52.627-0300 I CONTROL [main] Trying to start Windows service 'MongoDBDigisat'
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] MongoDB starting : pid=11908 port=12220 dbpath=c:/DigiSat/SuiteG6/Dados 64-bit host=programacao08
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] db version v3.4.7-rc0
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] allocator: tcmalloc
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] modules: none
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] build environment:
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] distarch: x86_64
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] target_arch: x86_64
2018-07-02T07:52:52.628-0300 I CONTROL [initandlisten] options: { config: "C:\DigiSat\SuiteG6\MongoDB\mongod.conf", net:

{ port: 12220 }

, processManagement: { windowsService:

{ description: "Service database Digisat Tecnologia", displayName: "MongoDB Digisat", serviceName: "MongoDBDigisat" }

}, replication: { replSetName: "rs0" }, security: { authorization: "enabled" }, service: true, setParameter: { enableLocalhostAuthBypass: "1" }, storage: { dbPath: "c:/DigiSat/SuiteG6/Dados/", journal:

{ enabled: true }

}, systemLog: { destination: "file", path: "c:/DigiSat/SuiteG6/MongoDB/log/mongo.log", quiet: true } }
2018-07-02T07:52:52.629-0300 I - [initandlisten] Detected data files in c:/DigiSat/SuiteG6/Dados created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-07-02T07:52:52.629-0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3526M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-07-02T07:54:25.733-0300 E STORAGE [initandlisten] WiredTiger error (0) [1530528865:732828][11908:140718538446544], file:collection-4--4661076328605078490.wt, txn-recover: read checksum error for 4096B block at offset 6307840: block header checksum of 1117133235 doesn't match expected checksum of 2785889967
2018-07-02T07:54:25.733-0300 E STORAGE [initandlisten] WiredTiger error (0) [1530528865:732828][11908:140718538446544], file:collection-4--4661076328605078490.wt, txn-recover: collection-4--4661076328605078490.wt: encountered an illegal file format or internal value
2018-07-02T07:54:25.733-0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1530528865:732828][11908:140718538446544], file:collection-4--4661076328605078490.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-07-02T07:54:25.733-0300 I - [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
2018-07-02T07:54:25.733-0300 I - [initandlisten]

***aborting after fassert() failure

2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d2c73
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d0964
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d01e5
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] ucrtbase.dll raise+0x1e7
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] ucrtbase.dll abort+0x31
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x160fa1
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d5c5
2018-07-02T07:54:25.742-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48b3c6
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48afb2
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48b66e
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3ff502
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fcbb5
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fd1b4
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fa8c1
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fdce1
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x40af21
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x42df13
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a253
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a315
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a424
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x43989d
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x485bc7
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48839b
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499692
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499f6e
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499b88
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x453f2f
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x49a5b4
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4316d0
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x42b30b
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1962
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1cf
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe ???
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4b497b
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4b6a77
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1a0c13
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] sechost.dll LsaFreeMemory+0x1c2
2018-07-02T07:54:25.743-0300 I CONTROL [initandlisten] KERNEL32.DLL BaseThreadInitThunk+0x14
2018-07-02T07:54:25.743-0300 F - [initandlisten] Got signal: 22 (SIGABRT).
2018-07-02T07:54:25.744-0300 I CONTROL [initandlisten] *** unhandled exception 0x0000000E at 0x00007FFB93183C58, terminating
2018-07-02T07:54:25.744-0300 I CONTROL [initandlisten] *** stack trace for unhandled exception:
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] KERNELBASE.dll RaiseException+0x68
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d01fd
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] ucrtbase.dll raise+0x1e7
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] ucrtbase.dll abort+0x31
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x160fa1
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1d5c5
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48b3c6
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48afb2
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48b66e
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3ff502
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fcbb5
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fd1b4
2018-07-02T07:54:25.746-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fa8c1
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x3fdce1
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x40af21
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x42df13
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a253
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a315
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48a424
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x43989d
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x485bc7
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x48839b
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499692
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499f6e
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x499b88
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x453f2f
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x49a5b4
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4316d0
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x42b30b
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1962
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1cf
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe ???
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4b497b
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe log1p+0x4b6a77
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] mongod.exe index_collator_extension+0x1a0c13
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] sechost.dll LsaFreeMemory+0x1c2
2018-07-02T07:54:25.747-0300 I CONTROL [initandlisten] KERNEL32.DLL BaseThreadInitThunk+0x14
2018-07-02T07:54:25.747-0300 I - [initandlisten]
2018-07-02T07:54:25.748-0300 I CONTROL [initandlisten] writing minidump diagnostic file C:\DigiSat\SuiteG6\MongoDB\bin\mongod.2018-07-02T10-54-25.mdmp
2018-07-02T07:54:25.808-0300 I CONTROL [initandlisten] *** immediate exit due to unhandled exception

Participants:

 Description   

I am attempting to start a database that was stopped unexpectedly and I receive the following error. The issue appears to be the with .wt file, but I have no way to repair it. 

WiredTiger.wt and WiredTiger.turtle attached.



 Comments   
Comment by Diego Pilger [ 03/Jul/18 ]

Thank you, Kelsey.

Comment by Kelsey Schubert [ 03/Jul/18 ]

Hi diegojoel10,

Unfortunately, this error indicates that there was corruption on the disk, most often caused by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

  • Make sure your underlying storage is configured in an optimal way.
  • Schedule and perform regular checks of the integrity of your filesystems and disks.
  • Make sure to update MongoDB to the most recent version. Note that versions ending with -rcX (like 3.4.7-rc0) are release candidates and are not recommended for use in production.
  • Never manipulate the underlying database files in any way while mongod is running.
  • Always keep up to date backups of your databases and verify that you have a process in place to restore them.
  • Use a replica set for improved reliability.

Thank you,
Kelsey 

Generated at Thu Feb 08 04:41:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.