WiredTiger fails to recover data when repairing the corrupted database

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: 3.6.18
    • Component/s: None
    • None
    • ALL
    • Hide

      Improper shutdown causes data corruption and the MongoDB service cannot start properly.

      Show
      Improper shutdown causes data corruption and the MongoDB service cannot start properly.
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      We are using MongoDB v3.6.18 as docker containers to store the logs from the applications. There are 3 instances of MongoDB running at any point of time.

      Today, the database got corrupted due to improper shutdown. When I tried to repair this, the repair tool gave an error saying that the operation is unsuccessful. The logs for the repair operation are follows:

      Repair

      [root@dcvsl125 mongodb]# docker run -it -v /docker/services/mongodb/db01:/data/db mongo:3.6.18 mongod --repair
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=c4feb442a438
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] db version v3.6.18
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] git version: 2005f25eed7ed88fa698d9b800fe536bb0410ba4
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] allocator: tcmalloc
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] modules: none
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] build environment:
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distmod: ubuntu1604
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distarch: x86_64
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] target_arch: x86_64
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] options: { net:

      Unknown macro: { bindIpAll}

      , repair: true }
      2021-12-07T01:49:59.778+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2021-12-07T01:49:59.778+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2021-12-07T01:49:59.779+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
      2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=63873M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
      2021-12-07T01:50:00.410+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
      2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] WiredTiger metadata corruption detected
      2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
      2021-12-07T01:50:00.414+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
      2021-12-07T01:50:00.414+0000 F - [initandlisten]

      ***aborting after fassert() failure

      [root@dcvsl125 mongodb]#

      Looking around for the issue, I came across the following Jira ticket (https://jira.mongodb.org/browse/SERVER-40088) which had the similar issue reported.

      Attaching the files from this database. 

       

      Can you please help me in recovering the data for this database?
      Please let me know if you need more information.

        1. WiredTiger.turtle
          1 kB
        2. WiredTiger.wt
          640 kB
        3. WiredTigerLAS.wt
          4 kB

            Assignee:
            Unassigned
            Reporter:
            Sarojini Jillalla
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: