[SERVER-49492] WiredTiger metadata corruption detected Created: 14/Jul/20  Updated: 27/Oct/23  Resolved: 28/Jul/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.12
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bora Yüret Assignee: Dmitry Agranat
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

After a server crash, we can't open our MongoDB instance. This is the status of the service:

root@xyz:~# systemctl status mongodsasshard.service
● mongodsasshard.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongodsasshard.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-07-14 13:33:11 +03; 2h 53min ago
Docs: https://docs.mongodb.org/manual
Process: 38452 ExecStart=/usr/bin/mongod --config /etc/mongodsasshard.conf (code=exited, status=14)
Main PID: 38452 (code=exited, status=14)

Jul 14 13:33:10 xyz systemd[1]: Started MongoDB Database Server.
Jul 14 13:33:10 xyz mongod[38452]: 2020-07-14T13:33:10.406+0300 I STORAGE [main] Max cache overflow file size custom option: 0
Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Main process exited, code=exited, status=14/n/a
Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Failed with result 'exit-code'.

 

 

And this is from the end of the log file:

 

2020-07-14T13:33:10.406+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2020-07-14T13:33:10.409+0300 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] MongoDB starting : pid=38452 port=27022 dbpath=/data/sas/MongoSasShardData 64-bit host=somtr
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] db version v4.0.12
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] git version: 5776e3cbf9e7afe86e6b29e22520ffb6766e95d4
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] modules: none
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] build environment:
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distmod: ubuntu1804
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distarch: x86_64
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] target_arch: x86_64
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] options: { config: "/etc/mongodsasshard.conf", net:

{ bindIp: "0.0.0.0", port: 27022 }

, replication: { replSetName: "rsSas" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/sas/MongoSasShardData", engine: "wiredTiger", journal:

{ enabled: true }

, wiredTiger: { engineConfig:

{ cacheSizeGB: 32.0 }

} }, systemLog: { destination: "file", logAppend: true, path: "/data/sas/log/mongodsasshard.log" } }
2020-07-14T13:33:10.413+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=32768M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] WiredTiger metadata corruption detected
2020-07-14T13:33:11.214+0300 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
2020-07-14T13:33:11.214+0300 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 700
2020-07-14T13:33:11.214+0300 F - [initandlisten]

***aborting after fassert() failure

 

 

I can't open the server now. I need help.

 

Thank you,

 

Bora

 

 

 



 Comments   
Comment by Dmitry Agranat [ 28/Jul/20 ]

Hi,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 15/Jul/20 ]

Hi borayuret@gmail.com,

This error indicates a potential file permission issue:

2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory

Can you check the permission on these files and report back?

Thanks,
Dima

Generated at Thu Feb 08 05:19:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.