Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-49492

WiredTiger metadata corruption detected

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.12
    • Component/s: None
    • None
    • ALL
    • None
    • 0
    • None
    • None
    • None
    • None
    • None
    • None

      After a server crash, we can't open our MongoDB instance. This is the status of the service:

      root@xyz:~# systemctl status mongodsasshard.service
      ● mongodsasshard.service - MongoDB Database Server
      Loaded: loaded (/lib/systemd/system/mongodsasshard.service; enabled; vendor preset: enabled)
      Active: failed (Result: exit-code) since Tue 2020-07-14 13:33:11 +03; 2h 53min ago
      Docs: https://docs.mongodb.org/manual
      Process: 38452 ExecStart=/usr/bin/mongod --config /etc/mongodsasshard.conf (code=exited, status=14)
      Main PID: 38452 (code=exited, status=14)

      Jul 14 13:33:10 xyz systemd[1]: Started MongoDB Database Server.
      Jul 14 13:33:10 xyz mongod[38452]: 2020-07-14T13:33:10.406+0300 I STORAGE [main] Max cache overflow file size custom option: 0
      Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Main process exited, code=exited, status=14/n/a
      Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Failed with result 'exit-code'.

       

       

      And this is from the end of the log file:

       

      2020-07-14T13:33:10.406+0300 I CONTROL [main] ***** SERVER RESTARTED *****
      2020-07-14T13:33:10.409+0300 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] MongoDB starting : pid=38452 port=27022 dbpath=/data/sas/MongoSasShardData 64-bit host=somtr
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] db version v4.0.12
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] git version: 5776e3cbf9e7afe86e6b29e22520ffb6766e95d4
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] allocator: tcmalloc
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] modules: none
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] build environment:
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distmod: ubuntu1804
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distarch: x86_64
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] target_arch: x86_64
      2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] options: { config: "/etc/mongodsasshard.conf", net:

      { bindIp: "0.0.0.0", port: 27022 }

      , replication: { replSetName: "rsSas" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/sas/MongoSasShardData", engine: "wiredTiger", journal:

      { enabled: true }

      , wiredTiger: { engineConfig:

      { cacheSizeGB: 32.0 }

      } }, systemLog: { destination: "file", logAppend: true, path: "/data/sas/log/mongodsasshard.log" } }
      2020-07-14T13:33:10.413+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=32768M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
      2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
      2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
      2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
      2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
      2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
      2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
      2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
      2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
      2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
      2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
      2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory
      2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error
      2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted
      2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory
      2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory
      2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
      2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] WiredTiger metadata corruption detected
      2020-07-14T13:33:11.214+0300 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
      2020-07-14T13:33:11.214+0300 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 700
      2020-07-14T13:33:11.214+0300 F - [initandlisten]

      ***aborting after fassert() failure

       

       

      I can't open the server now. I need help.

       

      Thank you,

       

      Bora

       

       

       

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            borayuret@gmail.com Bora Yüret
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: