[SERVER-81164] Reduce execution slowness of the fcv_upgrade_downgrade passthrough suites Created: 18/Sep/23  Updated: 18/Jan/24  Resolved: 18/Jan/24

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.3.0-rc0, 7.2.0-rc2

Type: Improvement Priority: Major - P3
Reporter: Xuerui Fa Assignee: Adi Zaimi
Resolution: Fixed Votes: 0
Labels: sharding-nyc-subteam3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
Assigned Teams:
Sharding NYC
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.2
Participants:

 Description   

From a local PB, it seems the fcv_upgrade_downgrade... passthrough suites are taking a significant amount of time to complete. Looking at the JS hook, it seems we may be sleeping the thread for long durations while running the test, which will cause slowness. This is also run per test, which means that tests that previously took <1s to complete might take upwards of 20+s now.

We should reduce the amount of times we run setFCV and sleep per test, and we should also avoid running the hook on every test. In mongosync passthrough testing, we would execute the hook only once every n=15 or n=20 tests, which would lead to the same test coverage over time assuming randomness in test order.



 Comments   
Comment by Adi Zaimi [ 18/Jan/24 ]

Hook running in background was corrected to not sleep for extensive interval.

Comment by Adi Zaimi [ 21/Nov/23 ]

Yes I agree with judah.schvimer@mongodb.com that this is a test only and should not block 7.2, however I already merged the fix for this to 7.2 branch so either way we should be good.

Comment by Githook User [ 20/Nov/23 ]

Author:

{'name': 'Adi Zaimi', 'email': 'adizaimi@yahoo.com', 'username': 'adizaimi'}

Message: SERVER-81164 Reduce test duration by sleeping less in the bg js hook

Reduce the duration of sleep() in the run_fcv_upgrade_downgrade_background.js background hook,
which is used in the fcf_upgrade_downgrade suites, which will result in tests and consequently the suite
to take an acceptable time to complete (similar to other test suites).
Branch: v7.2
https://github.com/mongodb/mongo/commit/ebe878ab31471507b7a74b51c322c0b2cd333488

Comment by Adi Zaimi [ 13/Nov/23 ]

FTR, I did look at ways of having the python script better control the js hook:
1. Doing sleeps of 1 sec inside a for loop up to the required duration does not cause the js script to exit early (when the foreground test is done) because the script is not seem to receive any external signals from the python fixture.
2. Using TestData in the JS hook is possible but modifying that state is futile because changes dont persist (in order to modify the script to only do one fcv operation but it needs to remember what operation it ran last)
3. Sending in environmental variables may be tricky/useless because the part of the fixture that calls the js script in a loop is not exposed to be modified by fcv_upgrade_downgrade.py so we can send the next operation to run into the js.
4. Possibly we can rework fcv_upgrade_downgrade.py to send the fcv command itself instead of relying on the js script and that might be the most likely one to have success without modifying the fixture itself, however I wonder if it is worth pursuing at this point (since sleeping for 1s seems to contain the test to within a reasonable time).

Comment by Adi Zaimi [ 13/Nov/23 ]

I changed the sleep duration to be 1s (instead of 5s) so each test will now take a lot less time than before, and cumulative time is roughly similar to the duration of similar suites without the fcv upgrade downgrade hook:
ex: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_display_fcv_upgrade_downgrade_sharded_collections_jscore_passthrough_56d5b028429d0f812397f132113ed32f569cace8_23_11_11_03_11_59
it now takes 2h 34m 10s (cumulative), and 36m 48s (wall clock).

Comment by Githook User [ 10/Nov/23 ]

Author:

{'name': 'Adi Zaimi', 'email': 'adizaimi@yahoo.com', 'username': 'adizaimi'}

Message: SERVER-81164 Reduce test duration by sleeping less in the bg js hook

Reduce the duration of sleep() in the run_fcv_upgrade_downgrade_background.js background hook,
which is used in the fcf_upgrade_downgrade suites, which will result in tests and consequently the suite
to take an acceptable time to complete (similar to other test suites).
Branch: master
https://github.com/mongodb/mongo/commit/78f7981be1d7de23fea577cc8141dba62b59dbd5

Generated at Thu Feb 08 06:45:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.