[SERVER-33075] Evergreen tasks with tests that have only "silentfail" status are still marked as failure Created: 02/Feb/18  Updated: 06/Dec/22  Resolved: 23/Oct/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.6.0-rc1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: [DO NOT ASSIGN] Backlog - Decision Automation Group (DAG) (Inactive)
Resolution: Won't Fix Votes: 0
Labels: close
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by SERVER-31470 Support having resmoke.py run resourc... Closed
Assigned Teams:
Decision Automation Group
Operating System: ALL
Steps To Reproduce:

Apply the following patch to pretend jstests/core/all.js is unreliable and have it fail.

python buildscripts/evergreen_run_tests.py --suites=core jstests/core/all.js --tagFile=etc/test_lifecycle.yml --reportFile=report.json --patchBuild
echo $?

diff --git a/buildscripts/resmokeconfig/suites/core.yml b/buildscripts/resmokeconfig/suites/core.yml
index ae4b60dc3e..c336731195 100644
--- a/buildscripts/resmokeconfig/suites/core.yml
+++ b/buildscripts/resmokeconfig/suites/core.yml
@@ -8,7 +8,9 @@ executor:
   config:
     shell_options:
       readMode: commands
-      eval: load("jstests/libs/override_methods/detect_spawning_own_mongod.js");
+      eval: >-
+        load("jstests/libs/override_methods/detect_spawning_own_mongod.js");
+        throw new Error("Intentionally thrown");
   hooks:
   - class: ValidateCollections
     shell_options:
diff --git a/etc/test_lifecycle.yml b/etc/test_lifecycle.yml
index 6a5146c6a0..edb996ca3d 100644
--- a/etc/test_lifecycle.yml
+++ b/etc/test_lifecycle.yml
@@ -1,4 +1,7 @@
 # This file is used to tag JS tests that run under resmoke.py.
 # The content of this file lives in the https://github.com/mongodb/mongo-test-metadata repository.
 # It is fetched automatically as part of the compile task.
-selector: {}
+selector:
+  js_test:
+    jstests/core/all.js:
+    - unreliable

Participants:

 Description   

This issue was introduced by my changes from 046a5a0 as part of SERVER-31470 because resmoke.py exits with a non-zero return code if any tests failed (regardless of the evergreen_status it'll use in the report.json file).



 Comments   
Comment by Max Hirschhorn [ 11/Oct/18 ]

schwerin has expressed a dislike of using different shades of colors to distinguish between whether a failure is something an engineer needs to look at or not. We should maybe consider never doing this ticket and instead investing effort to reorganize how the information is displayed in the Evergreen UI so it appears in lists according to whether the test is unreliable, the test failure is a known issue, etc.

Comment by Max Hirschhorn [ 02/Feb/18 ]

yves.duhem, the simplest way to I can think to address this is by changing the definition of TestReport.wasSuccessful() so that Suite.return_code is never set to a non-zero value if the only failing tests have evergreen_status="silentfail". That is to say, something along the lines of the following, but we'd probably want to also cache the result to avoid rescanning the entire list of test results each time TestReport.wasSuccessful() is called in the Job._run() test loop.

def wasSuccessful(self):
    """
    Returns true if all tests executed successfully, or if only
    tests with a "silentfail" Evergreen status failed.
    """
 
    with self._lock:
        if self.num_failed == 0:
            return self.num_errored == self.num_interrupted == 0:
 
        for test_info in self.test_infos:
            if test_info.evergreen_status != "silentfail":
                return False
 
        return True

Generated at Thu Feb 08 04:32:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.