[SERVER-45383] Add multiversion build variant to burn_in_tags_gen task Created: 06/Jan/20  Updated: 29/Oct/23  Resolved: 16/Jan/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Improvement Priority: Major - P3
Reporter: Jason Chan Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Repl 2020-01-13, Repl 2020-01-27
Participants:

 Description   

Currently, the burn_in_multiversion_gen task only runs each test once. This can be problematic for test suites like replica_sets_multiversion that randomly assign a binary version to each node for each test run since we won't be immediately running all possible configurations in our test patches. We should repeat these tests in the burn_in_multiversion task to increase the chances that we will cover all problematic multiversion configurations for these tests.



 Comments   
Comment by Githook User [ 16/Jan/20 ]

Author:

{'name': 'Jason Chan', 'email': 'jason.chan@mongodb.com', 'username': 'jasonjhchan'}

Message: SERVER-45383 Add multiversion build variant to burn_in_tags_gen task
Branch: master
https://github.com/mongodb/mongo/commit/537f6ca20aa73315d9266895dc928823507e38a1

Comment by Jason Chan [ 08/Jan/20 ]

Yes, I expect that running them for 10 minutes should be sufficient since the multiversion suites should not take much longer to run than their non-multiversion counterparts.

Comment by Judah Schvimer [ 07/Jan/20 ]

That does sound good. IIUC this will make patch builds that run burn_in_tags (which is required before pushing) run changes in replica_sets_multiversion and sharding_multiversion. How many times will it run those? As many times as fit in 10 minutes as David says is normal?

Comment by Jason Chan [ 07/Jan/20 ]

I talked with david.bradford and robert.guo about this.
Currently, burn_in_multiversion task will run all generated multiversion passthroughs under all possible combinations. However, for the tasks that run with random binary version assignments (replica_sets_multiversion and sharding_multiversion), it is not currently possible to run them under burn_in_tests on Enterprise RHEL 6.2 because burn_in_tests.py currently doesn't support running tasks that aren't normally run on the same build variant.

Instead, it might be better to create a dedicated multiversion build variant and use the burn_in_tags feature to run burn_in_tests against that build variant. The burn_in_tags script is used to generate burn_in_tests for build_variants that are not part of the build being run. Even with this new dedicated build variant, we will continue to run multiversion tasks under the UBSAN, ASAN, and DEBUG build variants but can remove the multiversion tasks from the Enterprise Ubuntu 18.04 variant.

Does this sound good to you? judah.schvimer

Comment by David Bradford (Inactive) [ 07/Jan/20 ]

Do you know how long it takes to run the tests once? For burn_in_tests we have been executing each test in a loop for test for 10 minutes and we get however many execution we get. So something like that would probably be ok. If the amount of extra work is going to be significant, we would probably want to check with the evergreen team to ensure they have the capacity to handle it.

As far as which team is best positioned, I would say your team. I don't think our team is very familiar with any of the multiversion_gen code.

Comment by Judah Schvimer [ 07/Jan/20 ]

david.bradford, how many times do you think we could realistically run multiversion tests in burn_in_multiversion_gen? I'm interested in bumping the number pretty high so with high probability we test every combination. Also, who is best positioned to do this work, your team or ours?

Generated at Thu Feb 08 05:08:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.