WiredTiger metadata corruption detected - unable to repair

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 3.6.18
    • Component/s: None
    • None
    • ALL
    • Hide

      Improper shutdown of the service causes data corruption.

      Show
      Improper shutdown of the service causes data corruption.
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      We are using Graylog, Elasticsearch and MongoDB for logging and archiving. These apps are run as docker containers with 3 replicas on 3 RHEL servers. We are using MongoDB version 3.6.18
      Generally, the docker containers goes down and they are automatically brought up by the docker daemon. But sometimes, the shutdown is not proper and the data in MongoDB gets corrupted. Till now, we used to perform a repair and the data was able to be recovered successfully.

      Today, the repair is failing to recover the data. As such the MongoDB node is not able to be started properly, thereby causing the Graylog server to be down.

      This is the output from the repair operation.

      My title

      [root@dcvsl125 mongodb]# docker run -it -v /docker/services/mongodb/db01:/data/db mongo:3.6.18 mongod --repair
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=c4feb442a438
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] db version v3.6.18
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] git version: 2005f25eed7ed88fa698d9b800fe536bb0410ba4
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] allocator: tcmalloc
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] modules: none
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] build environment:
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distmod: ubuntu1604
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] distarch: x86_64
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] target_arch: x86_64
      2021-12-07T01:49:59.777+0000 I CONTROL [initandlisten] options: { net:

      Unknown macro: { bindIpAll}

      , repair: true }
      2021-12-07T01:49:59.778+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2021-12-07T01:49:59.778+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2021-12-07T01:49:59.779+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
      2021-12-07T01:49:59.779+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=63873M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
      2021-12-07T01:50:00.410+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1638841800:410928][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1638841800:411036][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1638841800:411063][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
      2021-12-07T01:50:00.411+0000 E STORAGE [initandlisten] WiredTiger error (0) [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1638841800:411082][1:0x7f27e7e59a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
      2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] WiredTiger metadata corruption detected
      2021-12-07T01:50:00.414+0000 F STORAGE [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
      2021-12-07T01:50:00.414+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
      2021-12-07T01:50:00.414+0000 F - [initandlisten]

      ***aborting after fassert() failure

      [root@dcvsl125 mongodb]#

       

      Attaching the files from this corrupted instance.
      WiredTiger.turtle WiredTiger.wt WiredTigerLAS.wt

       

      I have searched here and found the https://jira.mongodb.org/browse/SERVER-40088, which is almost similar to the issue I am facing currently.

      Can you please help me in recovering from this failure?

       

        1. image003.png
          image003.png
          0.8 kB
        2. WiredTiger.turtle
          1 kB
        3. WiredTiger.wt
          640 kB
        4. WiredTigerLAS.wt
          4 kB

            Assignee:
            Edwin Zhou
            Reporter:
            Sarojini Jillalla
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: