Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19274

DB fails to recover after system crash, with "No such file or directory"

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.1.5
    • Component/s: Storage, WiredTiger
    • Fully Compatible
    • ALL
    • Hide

      Using the powercycle.sh script with the following arguments:

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54  -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50

      Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client.

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr>  -M <mongo bin dir> -R -N 10 -l 50
      Show
      Using the powercycle.sh script with the following arguments: ./powertest.sh -d https: //fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54 -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50 Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client. ./powertest.sh -d https: //fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr> -M <mongo bin dir> -R -N 10 -l 50

      The powercycle test created a DB that was not recoverable:

      2015-07-02T15:10:36.727-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
2015-07-02T15:10:36.788-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-315/data/wiredTiger/mongod.lock is not empty.
2015-07-02T15:10:36.788-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2015-07-02T15:10:36.788-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861392][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: /home/jonathan/pt-315/data/wiredTiger/collection-181-5805430730046625089.wt: No such file or directory
2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861464][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: Operation failed during recovery: No such file or directory
2015-07-02T15:10:36.880-0400 I -        [initandlisten] Fatal assertion 28548 NoSuchKey Unable to find metadata for table:collection-175-5805430730046625089
2015-07-02T15:10:36.880-0400 I -        [initandlisten]

***aborting after fassert() failure

        1. logs.tar
          343 kB
          Jonathan Abrahams
        2. powertest.sh
          32 kB
          Jonathan Abrahams
        3. wiredTiger.taraa
          140.00 MB
          Jonathan Abrahams
        4. wiredTiger.tarab
          140.00 MB
          Jonathan Abrahams
        5. wiredTiger.tarac
          140.00 MB
          Jonathan Abrahams
        6. wiredTiger.tarad
          140.00 MB
          Jonathan Abrahams
        7. wiredTiger.tarae
          140.00 MB
          Jonathan Abrahams
        8. wiredTiger.taraf
          140.00 MB
          Jonathan Abrahams
        9. wiredTiger.tarag
          29.68 MB
          Jonathan Abrahams
        10. wiredTiger-afterrecovery.taraa
          140.00 MB
          Jonathan Abrahams
        11. wiredTiger-afterrecovery.tarab
          140.00 MB
          Jonathan Abrahams
        12. wiredTiger-afterrecovery.tarac
          140.00 MB
          Jonathan Abrahams
        13. wiredTiger-afterrecovery.tarad
          140.00 MB
          Jonathan Abrahams
        14. wiredTiger-afterrecovery.tarae
          140.00 MB
          Jonathan Abrahams
        15. wiredTiger-afterrecovery.taraf
          140.00 MB
          Jonathan Abrahams
        16. wiredTiger-afterrecovery.tarag
          29.14 MB
          Jonathan Abrahams
        17. wiredTiger-beforerecovery.taraa
          140.00 MB
          Jonathan Abrahams
        18. wiredTiger-beforerecovery.tarab
          140.00 MB
          Jonathan Abrahams
        19. wiredTiger-beforerecovery.tarac
          140.00 MB
          Jonathan Abrahams
        20. wiredTiger-beforerecovery.tarad
          140.00 MB
          Jonathan Abrahams
        21. wiredTiger-beforerecovery.tarae
          140.00 MB
          Jonathan Abrahams
        22. wiredTiger-beforerecovery.taraf
          140.00 MB
          Jonathan Abrahams
        23. wiredTiger-beforerecovery.tarag
          31.80 MB
          Jonathan Abrahams
        24. wiredTiger-firstrun.tar
          11 kB
          Jonathan Abrahams

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            jonathan.abrahams Jonathan Abrahams
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: