[SERVER-24924] Exceptions during resmoke.py execution do not cause task failures Created: 06/Jul/16  Updated: 25/Apr/18  Resolved: 18/Apr/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.7.6

Type: Bug Priority: Major - P3
Reporter: Kamran K. Assignee: Jonathan Abrahams
Resolution: Done Votes: 0
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-24923 TypeError when running unit tests Closed
is related to SERVER-34116 resmoke.py is silently not running no... Closed
is related to SERVER-32285 dbtests suite runs no tests in resmok... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: TIG 2018-04-23
Participants:

 Description   

The unittests in this job didn't run because of TypeErrors, but the task still succeeded.

We need to investigate why the TypeErrors didn't propagate and cause the task to fail.



 Comments   
Comment by Githook User [ 18/Apr/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'username': 'hptabster', 'name': 'Jonathan Abrahams'}

Message: SERVER-24924 Fix lint
Branch: master
https://github.com/mongodb/mongo/commit/980e650068c23c762c3662216277fad576a7e7f4

Comment by Githook User [ 18/Apr/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'username': 'hptabster', 'name': 'Jonathan Abrahams'}

Message: SERVER-24924 Fix Exceptions during resmoke.py execution do not cause task failures
Branch: master
https://github.com/mongodb/mongo/commit/f1dce2d1934052cbac4032d0c5833d3857a0cfb2

Comment by Githook User [ 17/Apr/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: Revert "SERVER-24924 Resmoke raises an exception if the number of tests run is less than expected for a successful suite execution"

This reverts commit 2b20c142c7e15cdad1ef609bcb5f14ba288dacea.
Branch: master
https://github.com/mongodb/mongo/commit/366f5aea022ea2eccbce5c253e813c1151d53fc1

Comment by Githook User [ 17/Apr/18 ]

Author:

{'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com', 'username': 'hptabster'}

Message: SERVER-24924 Resmoke raises an exception if the number of tests run is less than expected for a successful suite execution
Branch: master
https://github.com/mongodb/mongo/commit/2b20c142c7e15cdad1ef609bcb5f14ba288dacea

Comment by Max Hirschhorn [ 23/Mar/18 ]

I think we should make it so that if the TestReport instance says that no tests were run, but the list of test files (or the list of dbtest suites) was non-empty, then resmoke.py should raise an exception that it expected to run at least N tests but didn't. I don't see a reason why after test selection occurs that there shouldn't be a guarantee we have at least that many tests in the report.

Comment by Robert Guo (Inactive) [ 14/Jul/16 ]

Just to clarify the problem: exceptions in child threads are silently ignored. Resmoke currently catches them by having each failure be added explicitly to a test report. This could be problematic if a failure is unexpected and therefore not added to the test report. We want a fail-safe here to ensure that any uncaught exceptions are propagated back to the main thread. This can be done by subclassing the Thread class and checking for exceptions in the parent. Similar to what is done here. Any exception also needs to be added to the report, possibly under a new category of failures, so that --continueOnFailure can ignore it if necessary.

Comment by Kamran K. [ 14/Jul/16 ]

Reopening. Our test harness can't silently break like this.

Comment by Robert Guo (Inactive) [ 14/Jul/16 ]

After chatting with max.hirschhorn, I'm going to close this as won't fix. There're 5 lines of code that can cause this problem. Making the problem go away will require adding more than 5 lines of code, so its seems to not be worth it.

Here's the last of the 5 lines that can cause the problem: https://github.com/mongodb/mongo/blob/r3.3.9/buildscripts/resmokelib/testing/job.py#L82

Generated at Thu Feb 08 04:07:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.