Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26884

Support archiving data files in Evergreen on test failure

    XMLWordPrintable

    Details

      Description

      There are test failures that are difficult to investigate without having access to the generated data files (e.g. backup_restore.js, wt_unclean_shutdown.js). We should consider either (a) adding support for tasks defined in etc/evergreen.yml to upload data files to S3, or (b) defining a new task in etc/evergreen.yml that runs these specific tests.

      If we decide to add support for archiving data files in Evergreen without changing the resmoke.py suite definitions (option "a"), then we'll need a mechanism to upload data files immediately after any test failure. This may require the changes from EVG-694 to execute the s3.put command in a resmoke.py after_test hook. (It is unlikely we could get away with moving the data files to a separate directory and uploading all of them at end of the suite without potentially filling up the disks on our EC2 machines.) If we decide to define a new task for archiving data files in Evergreen (option "b"), then we could simply run it with using continue_on_failure=false.

      Note: The archiving of the FTDC data is currently flawed, as only the FTDC data from the mongod processes in the last test (not necessarily even a failure) is uploaded to S3.

      Technique currently available for patch builds

      diff --git a/etc/evergreen.yml b/etc/evergreen.yml
      index 79bed55..f4b896e 100644
      --- a/etc/evergreen.yml
      +++ b/etc/evergreen.yml
      @@ -286,7 +286,7 @@ functions:
                 extra_args="$extra_args --continueOnFailure"
               fi
       
      -        ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
      +        ! ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
       
         "run jstestfuzz":
           - command: shell.exec
      @@ -1212,8 +1212,24 @@ tasks:
         - func: "do setup"
         - func: "run tests"
           vars:
      -      resmoke_args: --suites=no_passthrough --storageEngine=wiredTiger
      -      run_multiple_jobs: true
      +      resmoke_args: --executor=no_passthrough jstests/noPassthrough/backup_restore.js --storageEngine=wiredTiger --wiredTigerEngineConfigString="verbose=[checkpoint]" --repeat=50
      +      continue_on_failure: false
      +  - command: archive.targz_pack
      +    params:
      +      target: "dbpath.tgz"
      +      source_dir: "/data/db"
      +      include:
      +      - "**"
      +  - command: s3.put
      +    params:
      +      aws_key: ${aws_key}
      +      aws_secret: ${aws_secret}
      +      local_file: dbpath.tgz
      +      remote_file: mongodb-mongo-master/${build_variant}/${revision}/no_passthrough/${task_id}-${execution}.tgz
      +      bucket: mciuploads
      +      permissions: public-read
      +      content_type: ${content_type|application/x-gzip}
      +      display_name: Files in dbpath - Execution ${execution}
       
       - <<: *task_template
         name: noPassthroughWithMongod
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jonathan.abrahams Jonathan Abrahams
              Reporter:
              max.hirschhorn Max Hirschhorn
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: