[SERVER-61937] WiredTiger fails to recover data when repairing the corrupted database Created: 07/Dec/21  Updated: 13/Dec/21  Resolved: 13/Dec/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.18
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sarojini Jillalla Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File WiredTigerLAS.wt    
Issue Links:
Duplicate
duplicates SERVER-61936 WiredTiger metadata corruption detect... Closed
Operating System: ALL
Steps To Reproduce:

Improper shutdown causes data corruption and the MongoDB service cannot start properly.

Participants:

 Description   

We are using MongoDB v3.6.18 as docker containers to store the logs from the applications. There are 3 instances of MongoDB running at any point of time.

Today, the database got corrupted due to improper shutdown. When I tried to repair this, the repair tool gave an error saying that the operation is unsuccessful. The logs for the repair operation are follows:

Repair

[root@dcvsl125 mongodb]# docker run -it -v /docker/services/mongodb/db01:/data/db mongo:3.6.18 mongod --repair
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=c4feb442a438
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] db version v3.6.18
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] git version: 2005f25eed7ed88fa698d9b800fe536bb0410ba4
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] allocator: tcmalloc
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] modules: none
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] build environment:
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distmod: ubuntu1604
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distarch: x86_64
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] target_arch: x86_64
2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] options: { net:

Unknown macro: { bindIpAll}

, repair: true }
2021-12-07T01:49:59.778+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2021-12-07T01:49:59.778+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-12-07T01:49:59.779+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=63873M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-12-07T01:50:00.410+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] WiredTiger metadata corruption detected
2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
2021-12-07T01:50:00.414+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
2021-12-07T01:50:00.414+0000 F - [initandlisten]

***aborting after fassert() failure

[root@dcvsl125 mongodb]#

Looking around for the issue, I came across the following Jira ticket (https://jira.mongodb.org/browse/SERVER-40088) which had the similar issue reported.

Attaching the files from this database. 

 

Can you please help me in recovering the data for this database?
Please let me know if you need more information.



 Comments   
Comment by Edwin Zhou [ 13/Dec/21 ]

Thank you for letting us know this ticket duplicates SERVER-61936. I will now close this ticket.

Comment by Sarojini Jillalla [ 07/Dec/21 ]

Sorry, this can be closed.
Created a duplicate by mistake. (Duplicate of https://jira.mongodb.org/browse/SERVER-61936)

Generated at Thu Feb 08 05:53:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.