[SERVER-45383] Add multiversion build variant to burn_in_tags_gen task Created: 06/Jan/20 Updated: 29/Oct/23 Resolved: 16/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 4.3.3 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Jason Chan | Assignee: | Jason Chan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Backwards Compatibility: | Fully Compatible |
| Sprint: | Repl 2020-01-13, Repl 2020-01-27 |
| Participants: |
| Description |
|
Currently, the burn_in_multiversion_gen task only runs each test once. This can be problematic for test suites like replica_sets_multiversion that randomly assign a binary version to each node for each test run since we won't be immediately running all possible configurations in our test patches. We should repeat these tests in the burn_in_multiversion task to increase the chances that we will cover all problematic multiversion configurations for these tests. |
| Comments |
| Comment by Githook User [ 16/Jan/20 ] |
|
Author: {'name': 'Jason Chan', 'email': 'jason.chan@mongodb.com', 'username': 'jasonjhchan'}Message: |
| Comment by Jason Chan [ 08/Jan/20 ] |
|
Yes, I expect that running them for 10 minutes should be sufficient since the multiversion suites should not take much longer to run than their non-multiversion counterparts. |
| Comment by Judah Schvimer [ 07/Jan/20 ] |
|
That does sound good. IIUC this will make patch builds that run burn_in_tags (which is required before pushing) run changes in replica_sets_multiversion and sharding_multiversion. How many times will it run those? As many times as fit in 10 minutes as David says is normal? |
| Comment by Jason Chan [ 07/Jan/20 ] |
|
I talked with david.bradford and robert.guo about this. Instead, it might be better to create a dedicated multiversion build variant and use the burn_in_tags feature to run burn_in_tests against that build variant. The burn_in_tags script is used to generate burn_in_tests for build_variants that are not part of the build being run. Even with this new dedicated build variant, we will continue to run multiversion tasks under the UBSAN, ASAN, and DEBUG build variants but can remove the multiversion tasks from the Enterprise Ubuntu 18.04 variant. Does this sound good to you? judah.schvimer |
| Comment by David Bradford (Inactive) [ 07/Jan/20 ] |
|
Do you know how long it takes to run the tests once? For burn_in_tests we have been executing each test in a loop for test for 10 minutes and we get however many execution we get. So something like that would probably be ok. If the amount of extra work is going to be significant, we would probably want to check with the evergreen team to ensure they have the capacity to handle it. As far as which team is best positioned, I would say your team. I don't think our team is very familiar with any of the multiversion_gen code. |
| Comment by Judah Schvimer [ 07/Jan/20 ] |
|
david.bradford, how many times do you think we could realistically run multiversion tests in burn_in_multiversion_gen? I'm interested in bumping the number pretty high so with high probability we test every combination. Also, who is best positioned to do this work, your team or ours? |