Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19274

DB fails to recover after system crash, with "No such file or directory"

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.1.5
    • Component/s: Storage, WiredTiger
    • Labels:
    • Fully Compatible
    • ALL
    • Hide

      Using the powercycle.sh script with the following arguments:

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54  -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50

      Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client.

      ./powertest.sh -d https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr>  -M <mongo bin dir> -R -N 10 -l 50
      Show
      Using the powercycle.sh script with the following arguments: ./powertest.sh -d https: //fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s 10.4.1.54 -C mpower -O output1 -S "-i /Users/jonathan/.ssh/mFi.pem admin@10.4.117.131" -R -N 10 -l 50 Note - If you need to run this on an AWS instance you would use it as follows. The -F option is the root of the mongo source on your client. ./powertest.sh -d https: //fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.1.5.tgz -r pt-315 -F ~/mongodb/mongo -s "-i <pem file> <ip addr> -M <mongo bin dir> -R -N 10 -l 50

      The powercycle test created a DB that was not recoverable:

      2015-07-02T15:10:36.727-0400 I CONTROL  [main] ***** SERVER RESTARTED *****
2015-07-02T15:10:36.788-0400 W -        [initandlisten] Detected unclean shutdown - /home/jonathan/pt-315/data/wiredTiger/mongod.lock is not empty.
2015-07-02T15:10:36.788-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2015-07-02T15:10:36.788-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861392][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: /home/jonathan/pt-315/data/wiredTiger/collection-181-5805430730046625089.wt: No such file or directory
2015-07-02T15:10:36.861-0400 E STORAGE  [initandlisten] WiredTiger (2) [1435864236:861464][2600:0x7f6b3b048b80], file:collection-181-5805430730046625089.wt: Operation failed during recovery: No such file or directory
2015-07-02T15:10:36.880-0400 I -        [initandlisten] Fatal assertion 28548 NoSuchKey Unable to find metadata for table:collection-175-5805430730046625089
2015-07-02T15:10:36.880-0400 I -        [initandlisten]

***aborting after fassert() failure

        1. logs.tar
          343 kB
        2. powertest.sh
          32 kB
        3. wiredTiger.taraa
          140.00 MB
        4. wiredTiger.tarab
          140.00 MB
        5. wiredTiger.tarac
          140.00 MB
        6. wiredTiger.tarad
          140.00 MB
        7. wiredTiger.tarae
          140.00 MB
        8. wiredTiger.taraf
          140.00 MB
        9. wiredTiger.tarag
          29.68 MB
        10. wiredTiger-afterrecovery.taraa
          140.00 MB
        11. wiredTiger-afterrecovery.tarab
          140.00 MB
        12. wiredTiger-afterrecovery.tarac
          140.00 MB
        13. wiredTiger-afterrecovery.tarad
          140.00 MB
        14. wiredTiger-afterrecovery.tarae
          140.00 MB
        15. wiredTiger-afterrecovery.taraf
          140.00 MB
        16. wiredTiger-afterrecovery.tarag
          29.14 MB
        17. wiredTiger-beforerecovery.taraa
          140.00 MB
        18. wiredTiger-beforerecovery.tarab
          140.00 MB
        19. wiredTiger-beforerecovery.tarac
          140.00 MB
        20. wiredTiger-beforerecovery.tarad
          140.00 MB
        21. wiredTiger-beforerecovery.tarae
          140.00 MB
        22. wiredTiger-beforerecovery.taraf
          140.00 MB
        23. wiredTiger-beforerecovery.tarag
          31.80 MB
        24. wiredTiger-firstrun.tar
          11 kB

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            jonathan.abrahams Jonathan Abrahams
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: