Support archiving data files in Evergreen on test failure

XMLWordPrintableJSON

    • Fully Compatible
    • v3.6, v3.4
    • TIG 2018-02-12
    • 15
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      There are test failures that are difficult to investigate without having access to the generated data files (e.g. backup_restore.js, wt_unclean_shutdown.js). We should consider either (a) adding support for tasks defined in etc/evergreen.yml to upload data files to S3, or (b) defining a new task in etc/evergreen.yml that runs these specific tests.

      If we decide to add support for archiving data files in Evergreen without changing the resmoke.py suite definitions (option "a"), then we'll need a mechanism to upload data files immediately after any test failure. This may require the changes from EVG-694 to execute the s3.put command in a resmoke.py after_test hook. (It is unlikely we could get away with moving the data files to a separate directory and uploading all of them at end of the suite without potentially filling up the disks on our EC2 machines.) If we decide to define a new task for archiving data files in Evergreen (option "b"), then we could simply run it with using continue_on_failure=false.

      Note: The archiving of the FTDC data is currently flawed, as only the FTDC data from the mongod processes in the last test (not necessarily even a failure) is uploaded to S3.

      Technique currently available for patch builds
      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/etc/evergreen.yml b/etc/evergreen.yml
index 79bed55..f4b896e 100644
--- a/etc/evergreen.yml
+++ b/etc/evergreen.yml
@@ -286,7 +286,7 @@ functions:
           extra_args="$extra_args --continueOnFailure"
         fi
 
-        ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
+        ! ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
 
   "run jstestfuzz":
     - command: shell.exec
@@ -1212,8 +1212,24 @@ tasks:
   - func: "do setup"
   - func: "run tests"
     vars:
-      resmoke_args: --suites=no_passthrough --storageEngine=wiredTiger
-      run_multiple_jobs: true
+      resmoke_args: --executor=no_passthrough jstests/noPassthrough/backup_restore.js --storageEngine=wiredTiger --wiredTigerEngineConfigString="verbose=[checkpoint]" --repeat=50
+      continue_on_failure: false
+  - command: archive.targz_pack
+    params:
+      target: "dbpath.tgz"
+      source_dir: "/data/db"
+      include:
+      - "**"
+  - command: s3.put
+    params:
+      aws_key: ${aws_key}
+      aws_secret: ${aws_secret}
+      local_file: dbpath.tgz
+      remote_file: mongodb-mongo-master/${build_variant}/${revision}/no_passthrough/${task_id}-${execution}.tgz
+      bucket: mciuploads
+      permissions: public-read
+      content_type: ${content_type|application/x-gzip}
+      display_name: Files in dbpath - Execution ${execution}
 
 - <<: *task_template
   name: noPassthroughWithMongod

            Assignee:
            Jonathan Abrahams (Inactive)
            Reporter:
            Max Hirschhorn
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: