[SERVER-46662] Add some multiversion suites to a required builder Created: 05/Mar/20  Updated: 29/Oct/23  Resolved: 16/Mar/20

Status: Closed
Project: Core Server
Component/s: Replication, Testing Infrastructure, Upgrade/Downgrade
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Task Priority: Major - P3
Reporter: Judah Schvimer Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2020-03-12 at 5.48.44 PM.png     PNG File Screen Shot 2020-03-12 at 5.48.54 PM.png    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: Repl 2020-03-23
Participants:

 Comments   
Comment by Githook User [ 26/Mar/20 ]

Author:

{'name': 'Jason Chan', 'username': 'jasonjhchan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-46662 Add replica_sets_multiversion and sharding_multiversion to !Enterprise RHEL 6.2

(cherry picked from commit 304a242aa85bd70a3d902b5aacf7a6e4c4ac7df4)
Branch: v4.4
https://github.com/mongodb/mongo/commit/5599378dc4137896c87ec7104a9b1fe9917f9795

Comment by Githook User [ 16/Mar/20 ]

Author:

{'username': 'jasonjhchan', 'name': 'Jason Chan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-46662 Add replica_sets_multiversion and sharding_multiversion to !Enterprise RHEL 6.2
Branch: master
https://github.com/mongodb/mongo/commit/304a242aa85bd70a3d902b5aacf7a6e4c4ac7df4

Comment by William Schultz (Inactive) [ 12/Mar/20 ]

To give some examples from the safe reconfig project, here are a few BFs/bugs that were caught by replica_sets_multiversion and sharding_multiversion suites.

The first two BFs only appeared after a bug was pushed to master, so greater visibility of these failures on a required builder may have helped prevent the bugs from being pushed in the first place. The last ticket was able to address a bug caught in the replica_sets_multiversion suite that turned up before pushing. When making only C++ changes that don't trigger burn in tests, though, an engineer has to remember to include the right multiversion suites in their patch build, which I expect is more unlikely for a non replication engineer who is less familiar with those suites.

Also, to give a rough sense of how often these multiversion suites have been failing, below are links to the results of the replica_sets suite on Enterprise RHEL 6.2 required variant and the replica_sets_multiversion suite on Enterprise Ubuntu 18.04 variant over the last 1000 commits and 500 commits, respectively. The date ranges aren't exactly the same since we don't run replica_sets_multiversion suites as often, but it gives a sense of the chronic redness of the multiversion suite:

replica_sets

replica_sets_multiversion

Even setting aside the redness issue, I think it's sensible to include the replica_sets_multiversion and sharding_multiversion suites on a required builder, since they appear to be the suites more likely to catch replication related multiversion bugs that we introduce and putting them on a non-required builder makes it more likely engineers will forget to include them in patch builds or let the failures go unnoticed.

Comment by Judah Schvimer [ 09/Mar/20 ]

We think the replication and sharding jscore passthroughs would be the best candidates since those are deterministic. They would add a lot of time to the build and would not have caught the BFs we've seen, so we don't think these are worth it.

The replsets/ and sharding/ passthroughs are randomized, but we still think they are worth running to prevent build failures.

Generated at Thu Feb 08 05:12:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.