[SERVER-82200] burn_in_tags should include indirectly affected tests Created: 14/Oct/23  Updated: 29/Jan/24  Resolved: 15/Nov/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Tommaso Tocci Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File from_jstest_files_to_their_dependants_map.json     File generated-burn-in-config-653f6f0e0ae606a657a5dffa.tgz    
Issue Links:
Related
related to SERVER-85790 Innocuous change in jstests/libs/opti... Closed
related to SERVER-85858 Revert burn_in_test's inclusion of tr... Closed
Assigned Teams:
Correctness
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Currently, burn_in_tags only run js tests that have been explicitly modified in the patch.

On the other hand, if the patch affects a base FSM test, brun_in_tags will only run that base FSM and will not test all the FSM test derived from that one.

We have a similar issue also with js library modifications, in fact when a patch impacts a js test library, burn_in_tags will not execute any tests that use/include that library.

To improve js dependency discovery, we could potentially use the ES modules introduced in PM-3399



 Comments   
Comment by Githook User [ 29/Jan/24 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: Revert "SERVER-82200 Include indirectly affected tests for burn-in"

This reverts commit b9caeab9c76ea36e704571d1648e362caf4f7002.

GitOrigin-RevId: 6fef8824d84b67e6fe05070492a1b7f313a5f432
Branch: master
https://github.com/mongodb/mongo/commit/90b40b58dc32403996649baf1f54a604d1e3b657

Comment by Githook User [ 29/Jan/24 ]

Author:

{'name': 'Steve Gross', 'email': 'steve.gross@mongodb.com', 'username': 'stevegrossmongodb'}

Message: Revert "SERVER-82200 Update burn-in docs"

This reverts commit ef1b0924376fd0a0d563bc5050f037a70a87acfe.

GitOrigin-RevId: d20a53ae1f488f29598715139b6ea6df48d43085
Branch: master
https://github.com/mongodb/mongo/commit/20fdd22b71247138ea7e88f0fab3648b022fdf12

Comment by Githook User [ 14/Nov/23 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-82200 Update burn-in docs
Branch: master
https://github.com/mongodb/mongo/commit/ef1b0924376fd0a0d563bc5050f037a70a87acfe

Comment by Githook User [ 13/Nov/23 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-82200 Include indirectly affected tests for burn-in
Branch: master
https://github.com/mongodb/mongo/commit/b9caeab9c76ea36e704571d1648e362caf4f7002

Comment by Alex Neben [ 01/Nov/23 ]

You are right that in a perfect world changing one file that is included everywhere shouldn't fail burn in. However, this is a major step in the right direction. In most cases this will be better. In a few (such as the case you presented) it will be exactly the same because burn in already wasn't running. Now it will just be red.

Comment by Mikhail Shchatko [ 31/Oct/23 ]

alex.neben@mongodb.com but in this case the user is changing just one file and there is no way to split it into several patches or something to overcome the issue. Does it make sense to tolerate burn-in failure in such case?

Comment by Alex Neben [ 30/Oct/23 ]

I don't think we should assume generate.tasks will ever be able to support so much. For something this prolific i think it could be ok to fail burn in. We basically tell the user. Hey this touches so many files we cannot possibly ensure this is safe.

Comment by Mikhail Shchatko [ 30/Oct/23 ]

While investigating this I've got from_jstest_files_to_their_dependants_map.json, which shows that we have several files that have up to ~1700 dependant files, e.g. jstests/concurrency/fsm_workload_helpers/server_types.js, jstests/libs/uuid_util.js etc. It doesn't necessary means that there are 1700 tests that depend on a file with common code, some of them are also non-test files with common JS code, but there are definitely hundreds of tests that depend on those particular JS files. That means if someone changes that common file, burn-in-tests would want to generate variants with hundreds of tasks, because it generates a separate copy of the task for each affected test.

For example I have this patch, where I made a change to jstests/libs/uuid_util.js file, which generated a huge task generation config (26.7MB and 686185 lines of json) with hundreds of tasks (Generated Burn In Task Config - Execution 0 config file is attached to the task here and attached to this ticket generated-burn-in-config-653f6f0e0ae606a657a5dffa.tgz), which resulted in generate.tasks command failure here:

[2023/10/30 09:06:06.564] Command 'generate.tasks' in function 'generate version burn in' (step 9.7 of 9) failed: posting task data: sending generate.tasks request: server returned status 400: HTTP status code 400: 400 (Bad Request): unexpected end of JSON input.
[2023/10/30 09:06:06.564] Finished command 'generate.tasks' in function 'generate version burn in' (step 9.7 of 9) in 2m6.208040264s.
[2023/10/30 09:06:06.564] Running task commands failed: running command: command failed: posting task data: sending generate.tasks request: server returned status 400: HTTP status code 400: 400 (Bad Request): unexpected end of JSON input
[2023/10/30 09:06:06.564] Finished running task commands in 10m42.390088179s.

I think it blocks the solution to be merged until we make sure that generate.tasks command is able to consume such large configuration json and generate such large amount of tasks.

I opened a draft PR to come back to it later: https://github.com/10gen/mongo/pull/16499

CC steve.gross@mongodb.com alex.neben@mongodb.com

Generated at Thu Feb 08 06:48:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.