Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19274

DB fails to recover after system crash, with "No such file or directory"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1.5
    • Fix Version/s: None
    • Component/s: Storage, WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Using the powercycle.sh script with the following arguments:

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54  -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50
      

      Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client.

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr>  -M <mongo bin dir> -R -N 10 -l 50
      

      Show
      Using the powercycle.sh script with the following arguments: ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54 -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50 Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client. ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr> -M <mongo bin dir> -R -N 10 -l 50

      Description

      The powercycle test created a DB that was not recoverable:

      2015-07-02T15:10:36.727-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
      2015-07-02T15:10:36.788-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-315/data/wiredTiger/mongod.lock is not empty.
      2015-07-02T15:10:36.788-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-07-02T15:10:36.788-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861392][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: /home/jonathan/pt-315/data/wiredTiger/collection-181-5805430730046625089.wt: No such file or directory
      2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861464][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: Operation failed during recovery: No such file or directory
      2015-07-02T15:10:36.880-0400 I -        [initandlisten] Fatal assertion 28548 NoSuchKey Unable to find metadata for table:collection-175-5805430730046625089
      2015-07-02T15:10:36.880-0400 I -        [initandlisten]
       
      ***aborting after fassert() failure
      

        Attachments

        1. logs.tar
          343 kB
        2. powertest.sh
          32 kB
        3. wiredTiger.taraa
          140.00 MB
        4. wiredTiger.tarab
          140.00 MB
        5. wiredTiger.tarac
          140.00 MB
        6. wiredTiger.tarad
          140.00 MB
        7. wiredTiger.tarae
          140.00 MB
        8. wiredTiger.taraf
          140.00 MB
        9. wiredTiger.tarag
          29.68 MB
        10. wiredTiger-afterrecovery.taraa
          140.00 MB
        11. wiredTiger-afterrecovery.tarab
          140.00 MB
        12. wiredTiger-afterrecovery.tarac
          140.00 MB
        13. wiredTiger-afterrecovery.tarad
          140.00 MB
        14. wiredTiger-afterrecovery.tarae
          140.00 MB
        15. wiredTiger-afterrecovery.taraf
          140.00 MB
        16. wiredTiger-afterrecovery.tarag
          29.14 MB
        17. wiredTiger-beforerecovery.taraa
          140.00 MB
        18. wiredTiger-beforerecovery.tarab
          140.00 MB
        19. wiredTiger-beforerecovery.tarac
          140.00 MB
        20. wiredTiger-beforerecovery.tarad
          140.00 MB
        21. wiredTiger-beforerecovery.tarae
          140.00 MB
        22. wiredTiger-beforerecovery.taraf
          140.00 MB
        23. wiredTiger-beforerecovery.tarag
          31.80 MB
        24. wiredTiger-firstrun.tar
          11 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: