[SERVER-73566] Improve backports required exclusion mechanism/detection Created: 02/Feb/23  Updated: 16/Jan/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tausif Rahman (Inactive) Assignee: [DO NOT ASSIGN] Backlog - DevProd Correctness
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-72927 Get multiversion readme up to date Closed
depends on SERVER-72928 Expand burn-in-tags to all required &... Closed
depends on SERVER-72930 Create a multiversion linter Closed
Assigned Teams:
Correctness
Operating System: ALL
Participants:

 Description   

This is open ended. We should make this experience better. We probably need prevention of weird states by enforcing linting and improve test exclusion visibility so that devs can figure out what's running. Both of these are part of the overall project. Adding this ticket so that we keep this in mind during/after implementation.

NOTE: Also, it looks like this may have gotten messed up due to cleanup of "outdated" entries



 Comments   
Comment by Alex Neben [ 07/Feb/23 ]

FYI you can use memento in slack to capture slack messages like this

Comment by Tausif Rahman (Inactive) [ 02/Feb/23 ]

Randolph Tan
Does evergreen log something when it decides to exclude a test from running in a multiversion suite? I'm trying to figure out why a test is not being run in master when running in multiversion suite but gets in the multiversion suite of the older branch when backported

tausif.rahman
Is there an entry in this file? https://github.com/10gen/mongo/blob/b99588b40e4f5c44ed976438f58ce2249626204c/etc/backports_required_for_multiversion_tests.yml

maxh
(I feel like it'd help to have a link to a multiversion run where the test isn't being run for some commit and what the test expected to be included is)

Randolph Tan
master version, kind of old, https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_all_feature_flags_required_display_sharding_multiversion_patch_af1ad78010f1396ad09f2f96e16705598f45a54b_63d04d1a57e85a4c62fbeb63_23_01_24_21_27_11/execution-tasks?execution=0&sorts=STATUS%3AASC

Randolph Tan
backport version

Randolph Tan
https://spruce.mongodb.com/task/mongodb_mongo_v6.2_enterprise_rhel_80_64_bit_dynamic_required_sharding_last_continuous_2_linux_enterprise_patch_4fa7a897ed4677516bd6ef1e2f9d4298ec8b35de_63d98231850e61080e21a5a5_23_01_31_21_06_14/tests?execution=0&sortBy=STATUS&sortDir=ASC

Randolph Tan
both have entries on backports_required file, but I had to update the one on v6.2 branch with the new server ticket number because for some reason it is skipped in master
New

maxh
I don't see https://spruce.mongodb.com/version/63d98231850e61080e21a5a5/changes?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC changing the ticket number for the prepare_transaction_then_migrate.js entries in backports_required_for_multiversion_tests.yml. backports_required_for_multiversion_tests.yml for the 6.2 branch only has an entry for last-lts and doesn't have an entry for last-continuous

Randolph Tan
I did not change it here. I used this example only so it would be obviuos that it is being executed in the older branch

maxh
The rule is that both branches need the same file+ticket entry in the same section for the test to get run (edited)

Randolph Tan
But if when I look in master, it looks green because it did not get run

Randolph Tan
ah

Randolph Tan
because the older branch had the wrong entries it will never get run in master?

Randolph Tan
unless I 'fix' the older branch?

Randolph Tan
I thought it was based on the ticket number and it checks the git log

maxh
there's no SERVER-68361 entry on the 6.2 branch in last-continuous section so it won't run in sharding_multiversion_last_continuous and there is no SERVER-68361 sentry on the 6.0 branch in last-lts so it won't get run in sharding_multiversion_last_lts

maxh
I thought it was based on the ticket number and it checks the git log
Less clever than that. It relies on the byproduct of git cherry-pick making the same edits across the branches to that file

Randolph Tan
Hm... I see. I failed to update the ticket number when doing the original fix (twice actually from different tickets) because it didn't appear as a failure (probably because it was already not being run in master at that time). And the ticket number in the yml file is based on commit made in v5.1. I wonder if there's a better way of doing this so it would catch errors like this

maxh
It sounds like something would need to reconcile the Jira and/or git state to detect when an entry is omitted. Maybe
@tausif.rahman
can think on this with the rest of the SDP team. Perhaps there could be some kind of lint step when making changes on older branches which checks to see if there's an entry for the backported ticket on the later branch which hasn't been added to the older branch's backports_required_for_multiversion_tests.yml
:this:
1

Randolph Tan
Ok, I was so confused, but I think I know what happened

Randolph Tan
The entry was deleted for some reason

Randolph Tan
https://github.com/10gen/mongo/commit/7fbabf32d91ba028e6952b411aa2e7bfc253c342

maxh
heh must be a bug in garbage collection :kirby-sunglasses:

maxh
maybe there's a test case to turn this into for https://github.com/mongodb/mongo/blob/f8dba817139f2ea4493b6657f81e78cd8761b7c6/buildscripts/backports_required_for_multiversion_tests_deduplicator.py

Randolph Tan
If I understand it correctly, the ticket for cleaning up 'outdated' multiversion entry should have also deleted the entry in master

Randolph Tan
but it only deleted the one in v6.2 branch causing the 'anomaly'

tausif.rahman
I think we can improve this experience as part of PM-3104. I’ve created SERVER-73566 to track this

Generated at Thu Feb 08 06:25:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.