[SERVER-47827] Request for a local burn_in_tests.py that locally runs all tests changed since master, including those committed in the local branch Created: 13/Mar/20  Updated: 29/Oct/23  Resolved: 24/Jun/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.7.0, 4.4.2

Type: Bug Priority: Major - P3
Reporter: Maria van Keulen Assignee: Lydia Stepanek (Inactive)
Resolution: Fixed Votes: 1
Labels: tig-burnin
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
is related to SERVER-47230 burn_in_tests.py should default to RH... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: DAG 2020-05-04, DAG 2020-05-18, DAG 2020-06-29
Participants:
Story Points: 3

 Description   

When I attempt to run burn_in_tests.py locally, the wrong tests are chosen. Here is a snippet of the logs, showing only tests that are untracked in git:

[2020-03-13 17:20:15,455 - __main__ - DEBUG] 2020-03-13 17:20.15 Found changed tests            files={'jstests/replsets/oplog_sampling1.js', 'jstests/concurrency/fsm_workloads/collreindex.js', 'jstests/noPassthrough/duplicate_key_get_last_error.js', 'jstests/sharding/create_collection_prepared_transactions.js'}

Ideally, I would like local burn_in_tests.py to locally run all modified tests since master, including those that are already committed in the local branch, without needing to generate evergreen task configs.
It is my understanding that presently, local burn_in_tests.py will not run any tests that are committed in the local branch, even if they are not part of master.



 Comments   
Comment by Githook User [ 05/Oct/20 ]

Author:

{'name': 'Lydia Stepanek', 'email': 'lydia.stepanek@mongodb.com', 'username': 'lydiastepanek'}

Message: SERVER-47827 Request for a local burn_in_tests.py that locally runs all tests changed since master, including those committed in the local branch

(cherry picked from commit c848b16c777e78fa3b21d7b0ad538ee2072346d4)
Branch: v4.4
https://github.com/mongodb/mongo/commit/480579097465faffe5cbc4ce0ff002590be9bacf

Comment by David Bradford (Inactive) [ 25/Jun/20 ]

We can go back to 4.4, I've opened a BACKPORT ticket to do that.

There were pretty significant changes to burn_in_tests in 4.4, so going back further would be a good bit of work. So I'm going to hold off on that for now, but we could look backport all the improvements we have made at some point.

Comment by Bernard Gorman [ 24/Jun/20 ]

Love the new --origin-rev option! Could we possibly backport it at least to 4.4, and preferably further?

Comment by Githook User [ 24/Jun/20 ]

Author:

{'name': 'Lydia Stepanek', 'email': 'lydia.stepanek@mongodb.com', 'username': 'lydiastepanek'}

Message: SERVER-47827 Request for a local burn_in_tests.py that locally runs all tests changed since master, including those committed in the local branch
Branch: master
https://github.com/mongodb/mongo/commit/c848b16c777e78fa3b21d7b0ad538ee2072346d4

Comment by Lydia Stepanek (Inactive) [ 04/May/20 ]

https://mongodbcr.appspot.com/603970004/

Comment by David Bradford (Inactive) [ 17/Apr/20 ]

Here is the logic that `evergreen patch` uses to determine which files have changed. burn_in_test should attempt to do something similar.

A couple of notes:

  • Evergreen patch does not look at uncommitted changes by default. That logic is here. We probably should include uncommitted changes by default since that is the current behavior.
  • The logic for getting the diffs in used by a few places (burn_in, selected_tests, lint). And the git diff can look different in different situations (local runs, patch builds, commit-queue build, mainline builds). So we need to be careful that things don't break.
Comment by Maria van Keulen [ 27/Mar/20 ]

As an update, I found that all of the tests getting returned by burn_in_tests are untracked in git. I deleted these files and now burn_in_tests says "No new or modified tests found."

Comment by Maria van Keulen [ 27/Mar/20 ]

I tried this again, and got this result:
files=

{'jstests/core/txns/create_collection_write_conflict.js', 'jstests/noPassthrough/duplicate_key_get_last_error.js', 'jstests/replsets/oplog_sampling1.js', 'jstests/concurrency/fsm_workloads/collreindex.js', 'jstests/sharding/create_collection_prepared_transactions.js'}

Interestingly, the first test it finds doesn't exist in master; it's an uncommitted test from another branch of mine. Here is the diff of tests I've changed in my current branch since the base commit:

 jstests/concurrency/fsm_workloads/CRUD_and_commands.js                      |  76 +------------------------
 jstests/concurrency/fsm_workloads/CRUD_and_commands_with_createindexes.js   |  57 +++++++++++++++++++
 jstests/core/txns/commands_in_txns_read_concern.js                          | 110 ++++++++++++++++++++++++++++++++++++
 jstests/core/txns/create_collection.js                                      |   2 +-
 jstests/core/txns/create_collection_parallel.js                             |  28 ++++++++-
 jstests/core/txns/create_existing_indexes.js                                |   2 +-
 jstests/core/txns/create_indexes.js                                         |   2 +-
 jstests/core/txns/create_indexes_parallel.js                                |  34 ++++++++++-
 jstests/core/txns/implicit_collection_creation_in_txn.js                    |   5 +-
 .../noPassthrough/create_indexes_in_txn_errors_if_already_in_progress.js    |   2 +-

Comment by Brooke Miller [ 27/Mar/20 ]

Hey maria.vankeulen, which files were changed here? Looking to find more log information in order for us to reproduce this.

Comment by Maria van Keulen [ 13/Mar/20 ]

CC connie.chen

Generated at Thu Feb 08 05:15:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.