Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19274

DB fails to recover after system crash, with "No such file or directory"

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Cannot Reproduce
    • 3.1.5
    • None
    • Storage, WiredTiger
    • Fully Compatible
    • ALL
    • Hide

      Using the powercycle.sh script with the following arguments:

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54  -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50
      

      Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client.

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr>  -M <mongo bin dir> -R -N 10 -l 50
      

      Show
      Using the powercycle.sh script with the following arguments: ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54 -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50 Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client. ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr> -M <mongo bin dir> -R -N 10 -l 50

    Description

      The powercycle test created a DB that was not recoverable:

      2015-07-02T15:10:36.727-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
      2015-07-02T15:10:36.788-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-315/data/wiredTiger/mongod.lock is not empty.
      2015-07-02T15:10:36.788-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2015-07-02T15:10:36.788-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861392][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: /home/jonathan/pt-315/data/wiredTiger/collection-181-5805430730046625089.wt: No such file or directory
      2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861464][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: Operation failed during recovery: No such file or directory
      2015-07-02T15:10:36.880-0400 I -        [initandlisten] Fatal assertion 28548 NoSuchKey Unable to find metadata for table:collection-175-5805430730046625089
      2015-07-02T15:10:36.880-0400 I -        [initandlisten]
       
      ***aborting after fassert() failure
      

      Attachments

        1. powertest.sh
          32 kB
        2. logs.tar
          343 kB
        3. wiredTiger-firstrun.tar
          11 kB
        4. wiredTiger-afterrecovery.tarag
          29.14 MB
        5. wiredTiger-afterrecovery.tarad
          140.00 MB
        6. wiredTiger-afterrecovery.tarae
          140.00 MB
        7. wiredTiger-afterrecovery.taraf
          140.00 MB
        8. wiredTiger-afterrecovery.taraa
          140.00 MB
        9. wiredTiger-afterrecovery.tarab
          140.00 MB
        10. wiredTiger-afterrecovery.tarac
          140.00 MB
        11. wiredTiger.taraf
          140.00 MB
        12. wiredTiger.tarac
          140.00 MB
        13. wiredTiger.tarad
          140.00 MB
        14. wiredTiger.tarag
          29.68 MB
        15. wiredTiger.taraa
          140.00 MB
        16. wiredTiger.tarab
          140.00 MB
        17. wiredTiger.tarae
          140.00 MB
        18. wiredTiger-beforerecovery.tarad
          140.00 MB
        19. wiredTiger-beforerecovery.tarae
          140.00 MB
        20. wiredTiger-beforerecovery.taraf
          140.00 MB
        21. wiredTiger-beforerecovery.taraa
          140.00 MB
        22. wiredTiger-beforerecovery.tarab
          140.00 MB
        23. wiredTiger-beforerecovery.tarac
          140.00 MB
        24. wiredTiger-beforerecovery.tarag
          31.80 MB

        Issue Links

          Activity

            People

              sue.loverso@mongodb.com Susan LoVerso
              jonathan.abrahams Jonathan Abrahams
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: