[SERVER-72928] Expand burn-in-tags to all required & suggested variants Created: 17/Jan/23  Updated: 29/Oct/23  Resolved: 01/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0, 6.0.7, 7.0.0-rc3

Type: Task Priority: Major - P3
Reporter: Tausif Rahman (Inactive) Assignee: Tausif Rahman (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-73566 Improve backports required exclusion ... Backlog
Problem/Incident
Related
related to SERVER-80220 JavaScript tests tagged with in-devel... Closed
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.0, v6.0
Participants:
Linked BF Score: 160

 Description   
  • Create a new -dryRun diff resmoke command which tells you any tests your changes will start/stop running & run this command as part of every resmoke run in CI. This will make it easy for engineers to see how their changes affect the tests that are run. If -dryRun diff is giving unexpected results, it is almost certainly an infrastructure issue that needs to be addressed by SDP.
  • Appendix: Resmoke –-dryRun diff
  • Have a dry_run_multiversion_light_up variant which runs *dryRun diff* on every multiversion task. If there is any diff in the tests to run, it will fail the task & print the diff. This variant can be used as a quick way for engineers to see if the jsTest they added affects any multiversion tasks or not. It will not be a required variant, just something purely used for seeing what new tests will run on a per task basis.

We've looked into burn-in-tags & have decided this sends a better signal on whether a devs jstest changes will introduce multiversion failures (or other failures) or not. Instead, we've decided to expand burn-in-tags coverage to all required and suggested variants.

A separate task-generator ticket will be done in conjunction with this to make the UX more clear that jstests have been affected as a result of the current changes.



 Comments   
Comment by Githook User [ 01/Jun/23 ]

Author:

{'name': 'Tausif Rahman', 'email': 'tausif.rahman@mongodb.com', 'username': 'trahman1318'}

Message: SERVER-72928 Expand burn-in-tags to all required & suggested variants

(cherry picked from commit a4837acbd304b879a35026a34fcb9da6a96f2ef6)
(cherry picked from commit 71d027b52c27a615fdccbb1857af3665fd2c4e05)
Branch: v7.0
https://github.com/mongodb/mongo/commit/faa2a3b784ffff8cac7c52188f4aadc7d1399ce6

Comment by Githook User [ 01/Jun/23 ]

Author:

{'name': 'Tausif Rahman', 'email': 'tausif.rahman@mongodb.com', 'username': 'trahman1318'}

Message: SERVER-72928 Expand burn-in-tags to all required & suggested variants

(cherry picked from commit a4837acbd304b879a35026a34fcb9da6a96f2ef6)
(cherry picked from commit 71d027b52c27a615fdccbb1857af3665fd2c4e05)
Branch: v6.0
https://github.com/mongodb/mongo/commit/fcd109d4de5358ded0df7783d15178b2c6022ef1

Comment by Githook User [ 31/May/23 ]

Author:

{'name': 'Tausif Rahman', 'email': 'tausif.rahman@mongodb.com', 'username': 'trahman1318'}

Message: SERVER-72928: Use defaultdict after reading from cache
Branch: master
https://github.com/mongodb/mongo/commit/71d027b52c27a615fdccbb1857af3665fd2c4e05

Comment by Githook User [ 26/May/23 ]

Author:

{'name': 'Tausif Rahman', 'email': 'tausif.rahman@mongodb.com', 'username': 'trahman1318'}

Message: SERVER-72928 Expand burn-in-tags to all required & suggested variants
Branch: master
https://github.com/mongodb/mongo/commit/a4837acbd304b879a35026a34fcb9da6a96f2ef6

Comment by Annie Black [ 24/May/23 ]

That works. Thanks for taking a look!

Comment by Alex Neben [ 23/May/23 ]

I think the situation where we generate more than 2k tasks is very unlikely (but possible). I don't think we need to block this since the odds that someone does this are extremely slim. If someone does manage to do this in the next two weeks I will explain to them that they cannot run this burn in with 50k tasks. I am moving this to unblocked. Lmk if you feel strongly against this.

Comment by Tausif Rahman (Inactive) [ 23/May/23 ]

Okay, that sounds promising on our end w/ this ticket. We can very easily set an upper limit to the number of generated burn-in tasks that we activate. After EVG-19965 I think we'll have a pretty good idea.

Comment by Annie Black [ 23/May/23 ]

(We have actually noticed that 50,000 tasks can exist, they just can't all be activated. We'll look into understanding where we stop being able to activate tasks.)

Comment by Tausif Rahman (Inactive) [ 23/May/23 ]

annie.black@mongodb.com Sounds good! I'm okay with waiting on this & checking back in a week.

Comment by Annie Black [ 23/May/23 ]

tausif.rahman@mongodb.com we may need to block this work on EVG-19960 (planned for next week) – we discovered in EVG-19859 that projects that generate an extreme number of tasks (specifically we tested 50,000 tasks) can break the UI. If the work you have here is likely to generate many more tasks we'd suggest that you wait until this work is done to verify that patch pages are still possible. This week we're going to estimate our limit for tasks in EVG-19965 which should give us a better understanding.

cc alexander.neben@mongodb.com mohamed.khelif@mongodb.com  

Generated at Thu Feb 08 06:23:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.