[SERVER-40036] Database corruption, WiredTiger cannot be recovered with repair command Created: 08/Mar/19  Updated: 08/Mar/19  Resolved: 08/Mar/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aiden Wallis Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Tried to recover with mongod --repair and no luck.

 ```root@fossabot:/var/lib# mongod --repair --dbpath /var/lib/mongodb
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] MongoDB starting : pid=17227 port=27017 dbpath=/var/lib/mongodb 64-bit host=fossabot
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] db version v3.6.3
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] allocator: tcmalloc
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] modules: none
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] build environment:
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] distarch: x86_64
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] target_arch: x86_64
2019-03-08T10:20:12.868+0000 I CONTROL [initandlisten] options: { repair: true, storage:

{ dbPath: "/var/lib/mongodb" }

}
2019-03-08T10:20:12.868+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-03-08T10:20:12.868+0000 I STORAGE [initandlisten]
2019-03-08T10:20:12.868+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-03-08T10:20:12.868+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-03-08T10:20:12.868+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2019-03-08T10:20:12.868+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=31612M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-03-08T10:20:13.362+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1552040413:362092][17227:0x7fb694405f80], txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2019-03-08T10:20:13.362+0000 E STORAGE [initandlisten] WiredTiger error (0) [1552040413:362131][17227:0x7fb694405f80], txn-recover: WiredTiger is unable to read the recovery log.
2019-03-08T10:20:13.362+0000 E STORAGE [initandlisten] WiredTiger error (0) [1552040413:362152][17227:0x7fb694405f80], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2019-03-08T10:20:13.362+0000 E STORAGE [initandlisten] WiredTiger error (0) [1552040413:362160][17227:0x7fb694405f80], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2019-03-08T10:20:13.362+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1552040413:362170][17227:0x7fb694405f80], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2019-03-08T10:20:13.364+0000 E - [initandlisten] Assertion: 28718:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 389
2019-03-08T10:20:13.364+0000 I STORAGE [initandlisten] exception in initAndListen: Location28718: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2019-03-08T10:20:13.364+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-03-08T10:20:13.364+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-03-08T10:20:13.364+0000 I CONTROL [initandlisten] now exiting
2019-03-08T10:20:13.364+0000 I CONTROL [initandlisten] shutting down with code:100```



 Comments   
Comment by Danny Hatcher (Inactive) [ 08/Mar/19 ]

That's great to hear Aiden! Thanks for letting us know.

Danny

Comment by Aiden Wallis [ 08/Mar/19 ]

We found the issue, there was another process infinitely filling the hard drive. Our database after was restorable with the repair command, thanks!

Generated at Thu Feb 08 04:53:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.