[SERVER-43033] Corrupted WT files after power supply failure Created: 26/Aug/19  Updated: 27/Aug/19  Resolved: 27/Aug/19

Status: Closed
Project: Core Server
Component/s: Storage, WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ivan Makeev Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: wt-repair-success
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

db version v3.6.7
git version: 2628472127e9f1826e02c665c1d93880a204075e
OpenSSL version: OpenSSL 1.1.0f 25 May 2017
allocator: tcmalloc
modules: none
build environment:
distmod: debian92
distarch: x86_64
target_arch: x86_64


Attachments: File WiredTiger.turtle     File WiredTiger.wt    
Operating System: ALL
Participants:

 Description   

Hello. After power failure I cannot start mongod
Start with --repair flag also not helped.

First lines of log:

2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] MongoDB starting : pid=4661 port=27017 dbpath=/var/lib/mongodb 64-bit host=
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] db version v3.6.7
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0f 25 May 2017
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] allocator: tcmalloc
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] modules: none
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] build environment:
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] distmod: debian92
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] distarch: x86_64
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] target_arch: x86_64
2019-08-26T09:26:33.835+0300 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
2019-08-26T09:26:33.835+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2019-08-26T09:26:33.835+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-08-26T09:26:33.835+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-08-26T09:26:33.836+0300 I STORAGE [initandlisten] 
2019-08-26T09:26:33.836+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-08-26T09:26:33.836+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-08-26T09:26:33.836+0300 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2019-08-26T09:26:33.836+0300 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=15543M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-08-26T09:26:34.409+0300 E STORAGE [initandlisten] WiredTiger error (0) [1566800794:409510][4661:0x7f57ddef7580], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 28672B block at offset 221184: block header checksum of 1475890381 doesn't match expected checksum of 321370890
2019-08-26T09:26:34.409+0300 E STORAGE [initandlisten] WiredTiger error (0) [1566800794:409722][4661:0x7f57ddef7580], file:WiredTiger.wt, WT_CURSOR.insert: {221184, 28672, 321370890}: (chunk 1 of 28): 00 00 00 00 00 00 00 00 ac 35 19 00 00 00 00 00 25 6c 00 00 38 00 00 00 07 04 00 00 00 70 00 00 cd 4c f8 57 01 00 00 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 35 2d 2d 37 33 34 37 33 33 34 39 35 33 39 35 38 34 30 37 38 30 38 2e 77 74 00 80 c3 27 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 32 3d 28 61 64 64 72 3d 22 30 31 38 34 38 31 65 34 62 66 35 36 66 34 61 31 38 35 38 31 65 34 65 63 65 65 33 63 66 33 38 36 38 31 65 34 30 63 35 38 35 35 63 61 38 30 38 30 38 30 65 32 35 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 32 2c 74 69 6d 65 3d 31 35 33 36 38 35 31 31 39 38 2c 73 69 7a 65 3d 38 31 39 32 2c 77 72 69 74 65 5f 67 65 6e 3d 34 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 31 33 37 35 31 30 34 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 39 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 
....

Here is my WT and turtle files: WiredTiger.wt WiredTiger.turtle

Could you help me with restoring of DB? Thank you!



 Comments   
Comment by Danny Hatcher (Inactive) [ 27/Aug/19 ]

Great to hear! Thanks for updating us.

Comment by Ivan Makeev [ 27/Aug/19 ]

It was so simple Server is working now.

Thank you very much for your help!

Comment by Danny Hatcher (Inactive) [ 26/Aug/19 ]

Can you please try upgrading to the 4.0.12 binaries and then try the repair again? You should be able to repair with that version and then start up normally with your original 3.6 binaries.

Generated at Thu Feb 08 05:02:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.