[SERVER-57951] Incorrect data files archived by hook_test_archival.py Created: 22/Jun/21  Updated: 29/Oct/23  Resolved: 28/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.2, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Robert Guo (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: STM 2021-07-12
Participants:
Story Points: 2

 Description   

I ran a patch that failed with the expectation that the correct data files were archived, but looking at the task logs, this is what was archived:

[2021/06/17 19:49:10.810] [executor:js_test:job0] 19:49:10.810Z Archiving data files for test jstests/slow1/terminate_during_shutdown_checkpoint.js from /data/db/job0/resmoke
[2021/06/17 19:49:13.089] [executor:js_test:job0] 19:49:13.089Z Archive succeeded for jstests/slow1/terminate_during_shutdown_checkpoint.js: Tar/gzip Data files jstests/slow1/terminate_during_shutdown_checkpoint.js - Execution 0 Repetition 0 files: ['/data/db/job0/resmoke']

However, looking at the test logs, a different dbpath was used for the fixture (mongorunner/, not resmoke/):

[js_test:terminate_during_shutdown_checkpoint] ReplSetTest Starting....
[js_test:terminate_during_shutdown_checkpoint] Resetting db path '/data/db/job0/mongorunner/terminate_during_shutdown_checkpoint-0'
...
[js_test:terminate_during_shutdown_checkpoint] d20020| 2021-06-17T19:39:10.372+00:00 I  CONTROL  4615611 [initandlisten] "MongoDB starting","attr":{"pid":6176,"port":20020,"dbPath":"/data/db/job0/mongorunner/terminate_during_shutdown_checkpoint-0","architecture":"64-bit","host":"ip-10-122-89-72.ec2.internal"}

I had to apply the following change for this test to archive the correct data files:

diff --git a/buildscripts/resmokelib/testing/hook_test_archival.py b/buildscripts/resmokelib/testing/hook_test_archival.py
index fd0388be15..bae0051ada 100644
--- a/buildscripts/resmokelib/testing/hook_test_archival.py
+++ b/buildscripts/resmokelib/testing/hook_test_archival.py
@@ -93,7 +93,7 @@ class HookTestArchival(object):
                                                         config.EVERGREEN_EXECUTION,
                                                         self._tests_repeat[test_name])
         # Retrieve root directory for all dbPaths from fixture.
-        input_files = test.fixture.get_dbpath_prefix()
+        input_files = '/data/db/job0'
         s3_bucket = config.ARCHIVE_BUCKET
         s3_path = "{}/{}/{}/datafiles/{}".format(config.EVERGREEN_PROJECT_NAME,
                                                  config.EVERGREEN_VARIANT_NAME,

Not sure if this matters, but this test is in the slow1 suite.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 27/Jul/21 ]

Author:

{'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-57951 resmoke.py should archive all fixture locations for a job

(cherry picked from commit ffe55a9d08ecd09e70fb1298b3a37253a3be9cb4)
Branch: v5.0
https://github.com/mongodb/mongo/commit/6a10981efbef1337e4ab06e4f9c4454d1a333377

Comment by Githook User [ 27/Jun/21 ]

Author:

{'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-57951 resmoke.py should archive all fixture locations for a job
Branch: master
https://github.com/mongodb/mongo/commit/ffe55a9d08ecd09e70fb1298b3a37253a3be9cb4

Comment by Robert Guo (Inactive) [ 24/Jun/21 ]

gregory.wlodarek Thanks for digging into this. You're totally right that the issue is with the distinct fixtures. In general, we expect suites to either use shell or resmoke fixtures, but not both.

I've loosened the restriction for archival as part of this ticket, but there might be other limitations in the future if both are used.

Comment by Gregory Wlodarek [ 22/Jun/21 ]

I think this issue is caused because slow1 starts up its own fixture: https://github.com/mongodb/mongo/blob/da173c4995cd000da3d585c254f681f13edd0b2c/buildscripts/resmokeconfig/suites/slow1.yml#L18-L22.
So maybe there's an improvement we can make to the archiver to include any other data file paths used during the execution.

Generated at Thu Feb 08 05:43:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.