[SERVER-56793] running dbtest with no suites should be an error Created: 10/May/21  Updated: 29/Oct/23  Resolved: 09/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Eric Milkie Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-42265 dbtest fails when run without specify... Closed
is related to SERVER-57563 Make dbtest work as a unified process Closed
Backwards Compatibility: Fully Compatible
Sprint: STM 2021-06-14
Participants:
Story Points: 2

 Description   

Long ago, dbtest could be run with no args, and all the suites inside dbtest would run. This was how buildbot tested dbtest.
After the introduction of Evergreen, we switched the testing of dbtest to run each suite with a separate invocation of dbtest, in order to parallelize the run and make it faster. This means we lost test coverage of running dbtest with no arguments, and thus today dbtest actually fails if you run it with no arguments, since some tests leave behind state that cause subsequent tests to fail.
Since we have no plans to ever add back an Evergreen suite that runs dbtest with no arguments, we should make it an error to operate dbtest in this way, to prevent engineer confusion in the future. The error message should provide guidance as to how to run dbtest correctly (using resmoke will run dbtest separately for each suite, for example).



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 09/Jun/21 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-56793 running dbtest with no suites should be an error
Branch: master
https://github.com/mongodb/mongo/commit/46713337424f8d6c1541a5cb004a4364cf94f615

Comment by Louis Williams [ 04/Jun/21 ]

I don't know if waterfall and build failures are a good measure of the usefulness of dbtest. Like unit tests, dbtest is required for patch builds and are mostly deterministic. Commits with failures are often due to negligence or unexpected interactions with non-required builders (like ephemeralForTest).

I think we have quite a few dbtests that provide value. For example, proving that our WT timestamping logic works, and that our validate implementation works.

This is not to say that these tests need to stay in dbtest, as they could be moved to be unit tests, but I don't think we can delete these tests entirely. That timestamping test pulls in the entire system, so it would also be a hassle to migrate.

Comment by Andrew Morrow (Inactive) [ 04/Jun/21 ]
  • How many BFs resolved as "fixed" have been due to a waterfall failure in dbtest in the past year?
  • How many commit queue merge failures have been because of a failure in dbtest in the past year?
  • What percent of patch runs of dbtest have failed in the past year?

If we don't want to invest the energy to keep this monolith functioning correctly, I think we should consider whether it is actually delivering value. If not, perhaps we should delete it.

Comment by Eric Milkie [ 03/Jun/21 ]

robert.guo sorry I should have been more exact in my report; we need to error out if dbtest is started with nothing for "suite" in the following:

usage: build/install/bin/dbtest [options] [suite]...

(All the options start with a hyphen.) So I think you need to go to the place where the suites are parsed out of the command line and error out there instead.

Comment by Robert Guo (Inactive) [ 03/Jun/21 ]

mikhail.shchatko We can check if argc == 1 here and error out with a detailed message if so.

acm Feel free to set up something with interested audience to discuss the future of dbtest and include me. I don't have a lot of context at the moment on its status and history beyond what Eric said.

Comment by Andrew Morrow (Inactive) [ 11/May/21 ]

I dislike that we are in a position where the build system produces a binary which doesn't actually function. It means, for instance, that techniques like the prove- aliases cannot be applied to dbtest because it can't be executed as a whole.

  • Does dbtest still provide real value? Could we measure the differential coverage data between a run of the unit tests and a run of the unit tests and dbtest? Maybe we don't even need it anymore and we could just delete it.
  • If dbtest does still provide value, could we identify the bad tests and either fix them, or split them out into standalone dbtests?
  • Have we examined whether the current split-up of execution of dbtest really buys us anything? The tasks all seem to run in seconds. Would we even notice the difference if we just had resmoke run it as monolith rather than running multiple tasks? That would allow us to keep it functioning.
Generated at Thu Feb 08 05:40:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.