Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26884

Support archiving data files in Evergreen on test failure



    • Fully Compatible
    • v3.6, v3.4
    • TIG 2018-02-12
    • 15


      There are test failures that are difficult to investigate without having access to the generated data files (e.g. backup_restore.js, wt_unclean_shutdown.js). We should consider either (a) adding support for tasks defined in etc/evergreen.yml to upload data files to S3, or (b) defining a new task in etc/evergreen.yml that runs these specific tests.

      If we decide to add support for archiving data files in Evergreen without changing the resmoke.py suite definitions (option "a"), then we'll need a mechanism to upload data files immediately after any test failure. This may require the changes from EVG-694 to execute the s3.put command in a resmoke.py after_test hook. (It is unlikely we could get away with moving the data files to a separate directory and uploading all of them at end of the suite without potentially filling up the disks on our EC2 machines.) If we decide to define a new task for archiving data files in Evergreen (option "b"), then we could simply run it with using continue_on_failure=false.

      Note: The archiving of the FTDC data is currently flawed, as only the FTDC data from the mongod processes in the last test (not necessarily even a failure) is uploaded to S3.

      Technique currently available for patch builds

      diff --git a/etc/evergreen.yml b/etc/evergreen.yml
      index 79bed55..f4b896e 100644
      --- a/etc/evergreen.yml
      +++ b/etc/evergreen.yml
      @@ -286,7 +286,7 @@ functions:
                 extra_args="$extra_args --continueOnFailure"
      -        ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
      +        ! ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
         "run jstestfuzz":
           - command: shell.exec
      @@ -1212,8 +1212,24 @@ tasks:
         - func: "do setup"
         - func: "run tests"
      -      resmoke_args: --suites=no_passthrough --storageEngine=wiredTiger
      -      run_multiple_jobs: true
      +      resmoke_args: --executor=no_passthrough jstests/noPassthrough/backup_restore.js --storageEngine=wiredTiger --wiredTigerEngineConfigString="verbose=[checkpoint]" --repeat=50
      +      continue_on_failure: false
      +  - command: archive.targz_pack
      +    params:
      +      target: "dbpath.tgz"
      +      source_dir: "/data/db"
      +      include:
      +      - "**"
      +  - command: s3.put
      +    params:
      +      aws_key: ${aws_key}
      +      aws_secret: ${aws_secret}
      +      local_file: dbpath.tgz
      +      remote_file: mongodb-mongo-master/${build_variant}/${revision}/no_passthrough/${task_id}-${execution}.tgz
      +      bucket: mciuploads
      +      permissions: public-read
      +      content_type: ${content_type|application/x-gzip}
      +      display_name: Files in dbpath - Execution ${execution}
       - <<: *task_template
         name: noPassthroughWithMongod


        Issue Links



              jonathan.abrahams Jonathan Abrahams
              max.hirschhorn@mongodb.com Max Hirschhorn
              1 Vote for this issue
              3 Start watching this issue