[SERVER-26884] Support archiving data files in Evergreen on test failure Created: 02/Nov/16  Updated: 13/Jul/18  Resolved: 08/Feb/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.4.16, 3.6.6, 3.7.2

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Done Votes: 1
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-32898 Create unit test for resmoke module a... Closed
is depended on by SERVER-33144 Support archiving data files in Everg... Closed
is depended on by SERVER-33193 Enable test failure archive for speci... Closed
Related
related to SERVER-36111 Bug in job.py related to functionalit... Closed
related to SERVER-31562 dump replica set oplogs at the end of... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6, v3.4
Sprint: TIG 2018-02-12
Participants:
Linked BF Score: 15

 Description   

There are test failures that are difficult to investigate without having access to the generated data files (e.g. backup_restore.js, wt_unclean_shutdown.js). We should consider either (a) adding support for tasks defined in etc/evergreen.yml to upload data files to S3, or (b) defining a new task in etc/evergreen.yml that runs these specific tests.

If we decide to add support for archiving data files in Evergreen without changing the resmoke.py suite definitions (option "a"), then we'll need a mechanism to upload data files immediately after any test failure. This may require the changes from EVG-694 to execute the s3.put command in a resmoke.py after_test hook. (It is unlikely we could get away with moving the data files to a separate directory and uploading all of them at end of the suite without potentially filling up the disks on our EC2 machines.) If we decide to define a new task for archiving data files in Evergreen (option "b"), then we could simply run it with using continue_on_failure=false.

Note: The archiving of the FTDC data is currently flawed, as only the FTDC data from the mongod processes in the last test (not necessarily even a failure) is uploaded to S3.

Technique currently available for patch builds

diff --git a/etc/evergreen.yml b/etc/evergreen.yml
index 79bed55..f4b896e 100644
--- a/etc/evergreen.yml
+++ b/etc/evergreen.yml
@@ -286,7 +286,7 @@ functions:
           extra_args="$extra_args --continueOnFailure"
         fi
 
-        ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
+        ! ${path_prefix} ${asan_symbolizer} ${enable_lsan} ${rlp_environment} ${python|python} buildscripts/resmoke.py ${resmoke_args} $extra_args ${test_flags} --log=buildlogger --reportFile=report.json
 
   "run jstestfuzz":
     - command: shell.exec
@@ -1212,8 +1212,24 @@ tasks:
   - func: "do setup"
   - func: "run tests"
     vars:
-      resmoke_args: --suites=no_passthrough --storageEngine=wiredTiger
-      run_multiple_jobs: true
+      resmoke_args: --executor=no_passthrough jstests/noPassthrough/backup_restore.js --storageEngine=wiredTiger --wiredTigerEngineConfigString="verbose=[checkpoint]" --repeat=50
+      continue_on_failure: false
+  - command: archive.targz_pack
+    params:
+      target: "dbpath.tgz"
+      source_dir: "/data/db"
+      include:
+      - "**"
+  - command: s3.put
+    params:
+      aws_key: ${aws_key}
+      aws_secret: ${aws_secret}
+      local_file: dbpath.tgz
+      remote_file: mongodb-mongo-master/${build_variant}/${revision}/no_passthrough/${task_id}-${execution}.tgz
+      bucket: mciuploads
+      permissions: public-read
+      content_type: ${content_type|application/x-gzip}
+      display_name: Files in dbpath - Execution ${execution}
 
 - <<: *task_template
   name: noPassthroughWithMongod



 Comments   
Comment by Githook User [ 23/May/18 ]

Author:

{'username': 'hptabster', 'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com'}

Message: SERVER-26884 Support archiving data files in Evergreen on test failure
Branch: v3.6
https://github.com/mongodb/mongo/commit/90776222899f867288d9207edf67ca576f3d6dc2

Comment by Githook User [ 23/May/18 ]

Author:

{'username': 'hptabster', 'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com'}

Message: SERVER-26884 Support archiving data files in Evergreen on test failure
Branch: v3.4
https://github.com/mongodb/mongo/commit/75fc64f409e5eae456b7462d628b0460055c6f77

Comment by Githook User [ 08/Feb/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}

Message: SERVER-26884 Support archiving data files in Evergreen on test failure
Branch: master
https://github.com/mongodb/mongo/commit/4cc459b6eec57202943fb56d79327f5731f2bfef

Comment by Jonathan Abrahams [ 06/Feb/18 ]

The first phase of work will be on Linux.
Windows follow on work will be done on SERVER-33144

Comment by Jonathan Abrahams [ 25/Jan/18 ]

Changes to etc/evergreen,yml:

  • Remove gather and archive for FTDC data
  • Invoke resmoke.py with new parameters

Other changes:

  • Add boto3 to buildscripts/resmokelib/requirements.txt
Comment by Jonathan Abrahams [ 19/Jan/18 ]

Support to archive a test failure will be controlled as follows:

  • New entry in the resmokeconfig suite YML file to designate which tests or hooks should trigger an archive
  • New parameter to resmoke to archive a test on failure
Generated at Thu Feb 08 04:13:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.