[SERVER-46662] Add some multiversion suites to a required builder Created: 05/Mar/20 Updated: 29/Oct/23 Resolved: 16/Mar/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Testing Infrastructure, Upgrade/Downgrade |
| Affects Version/s: | None |
| Fix Version/s: | 4.4.0-rc0, 4.7.0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Judah Schvimer | Assignee: | Jason Chan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Backport Requested: |
v4.4
|
||||
| Sprint: | Repl 2020-03-23 | ||||
| Participants: | |||||
| Comments |
| Comment by Githook User [ 26/Mar/20 ] |
|
Author: {'name': 'Jason Chan', 'username': 'jasonjhchan', 'email': 'jason.chan@10gen.com'}Message: (cherry picked from commit 304a242aa85bd70a3d902b5aacf7a6e4c4ac7df4) |
| Comment by Githook User [ 16/Mar/20 ] |
|
Author: {'username': 'jasonjhchan', 'name': 'Jason Chan', 'email': 'jason.chan@10gen.com'}Message: |
| Comment by William Schultz (Inactive) [ 12/Mar/20 ] |
|
To give some examples from the safe reconfig project, here are a few BFs/bugs that were caught by replica_sets_multiversion and sharding_multiversion suites.
The first two BFs only appeared after a bug was pushed to master, so greater visibility of these failures on a required builder may have helped prevent the bugs from being pushed in the first place. The last ticket was able to address a bug caught in the replica_sets_multiversion suite that turned up before pushing. When making only C++ changes that don't trigger burn in tests, though, an engineer has to remember to include the right multiversion suites in their patch build, which I expect is more unlikely for a non replication engineer who is less familiar with those suites. Also, to give a rough sense of how often these multiversion suites have been failing, below are links to the results of the replica_sets suite on Enterprise RHEL 6.2 required variant and the replica_sets_multiversion suite on Enterprise Ubuntu 18.04 variant over the last 1000 commits and 500 commits, respectively. The date ranges aren't exactly the same since we don't run replica_sets_multiversion suites as often, but it gives a sense of the chronic redness of the multiversion suite:
Even setting aside the redness issue, I think it's sensible to include the replica_sets_multiversion and sharding_multiversion suites on a required builder, since they appear to be the suites more likely to catch replication related multiversion bugs that we introduce and putting them on a non-required builder makes it more likely engineers will forget to include them in patch builds or let the failures go unnoticed. |
| Comment by Judah Schvimer [ 09/Mar/20 ] |
|
We think the replication and sharding jscore passthroughs would be the best candidates since those are deterministic. They would add a lot of time to the build and would not have caught the BFs we've seen, so we don't think these are worth it. The replsets/ and sharding/ passthroughs are randomized, but we still think they are worth running to prevent build failures. |