-
Type:
Bug
-
Resolution: Gone away
-
Priority:
Major - P3
-
None
-
Affects Version/s: 4.0.12
-
Component/s: None
-
None
-
ALL
-
None
-
0
-
None
-
None
-
None
-
None
-
None
-
None
After a server crash, we can't open our MongoDB instance. This is the status of the service:
root@xyz:~# systemctl status mongodsasshard.service
● mongodsasshard.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongodsasshard.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-07-14 13:33:11 +03; 2h 53min ago
Docs: https://docs.mongodb.org/manual
Process: 38452 ExecStart=/usr/bin/mongod --config /etc/mongodsasshard.conf (code=exited, status=14)
Main PID: 38452 (code=exited, status=14)
Jul 14 13:33:10 xyz systemd[1]: Started MongoDB Database Server.
Jul 14 13:33:10 xyz mongod[38452]: 2020-07-14T13:33:10.406+0300 I STORAGE [main] Max cache overflow file size custom option: 0
Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Main process exited, code=exited, status=14/n/a
Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Failed with result 'exit-code'.
And this is from the end of the log file:
2020-07-14T13:33:10.406+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2020-07-14T13:33:10.409+0300 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] MongoDB starting : pid=38452 port=27022 dbpath=/data/sas/MongoSasShardData 64-bit host=somtr
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] db version v4.0.12
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] git version: 5776e3cbf9e7afe86e6b29e22520ffb6766e95d4
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] modules: none
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] build environment:
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distmod: ubuntu1804
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distarch: x86_64
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] target_arch: x86_64
2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] options: { config: "/etc/mongodsasshard.conf", net:
, replication: { replSetName: "rsSas" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/sas/MongoSasShardData", engine: "wiredTiger", journal:
{ enabled: true }, wiredTiger: { engineConfig:
{ cacheSizeGB: 32.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/data/sas/log/mongodsasshard.log" } }
2020-07-14T13:33:10.413+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=32768M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] WiredTiger metadata corruption detected
2020-07-14T13:33:11.214+0300 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
2020-07-14T13:33:11.214+0300 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 700
2020-07-14T13:33:11.214+0300 F - [initandlisten]
***aborting after fassert() failure
I can't open the server now. I need help.
Thank you,
Bora