Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33075

Evergreen tasks with tests that have only "silentfail" status are still marked as failure

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.0-rc1
    • Component/s: Testing Infrastructure
    • Labels:
    • Decision Automation Group
    • ALL
    • Hide

      Apply the following patch to pretend jstests/core/all.js is unreliable and have it fail.

      python buildscripts/evergreen_run_tests.py --suites=core jstests/core/all.js --tagFile=etc/test_lifecycle.yml --reportFile=report.json --patchBuild
      echo $?
      
      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/buildscripts/resmokeconfig/suites/core.yml b/buildscripts/resmokeconfig/suites/core.yml
      index ae4b60dc3e..c336731195 100644
      --- a/buildscripts/resmokeconfig/suites/core.yml
      +++ b/buildscripts/resmokeconfig/suites/core.yml
      @@ -8,7 +8,9 @@ executor:
         config:
           shell_options:
             readMode: commands
      -      eval: load("jstests/libs/override_methods/detect_spawning_own_mongod.js");
      +      eval: >-
      +        load("jstests/libs/override_methods/detect_spawning_own_mongod.js");
      +        throw new Error("Intentionally thrown");
         hooks:
         - class: ValidateCollections
           shell_options:
      diff --git a/etc/test_lifecycle.yml b/etc/test_lifecycle.yml
      index 6a5146c6a0..edb996ca3d 100644
      --- a/etc/test_lifecycle.yml
      +++ b/etc/test_lifecycle.yml
      @@ -1,4 +1,7 @@
       # This file is used to tag JS tests that run under resmoke.py.
       # The content of this file lives in the https://github.com/mongodb/mongo-test-metadata repository.
       # It is fetched automatically as part of the compile task.
      -selector: {}
      +selector:
      +  js_test:
      +    jstests/core/all.js:
      +    - unreliable
      
      Show
      Apply the following patch to pretend jstests/core/all.js is unreliable and have it fail. python buildscripts/evergreen_run_tests.py --suites=core jstests/core/all.js --tagFile=etc/test_lifecycle.yml --reportFile=report.json --patchBuild echo $? Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml diff --git a/buildscripts/resmokeconfig/suites/core.yml b/buildscripts/resmokeconfig/suites/core.yml index ae4b60dc3e..c336731195 100644 --- a/buildscripts/resmokeconfig/suites/core.yml +++ b/buildscripts/resmokeconfig/suites/core.yml @@ -8,7 +8,9 @@ executor: config: shell_options: readMode: commands - eval: load( "jstests/libs/override_methods/detect_spawning_own_mongod.js" ); + eval: >- + load( "jstests/libs/override_methods/detect_spawning_own_mongod.js" ); + throw new Error( "Intentionally thrown" ); hooks: - class: ValidateCollections shell_options: diff --git a/etc/test_lifecycle.yml b/etc/test_lifecycle.yml index 6a5146c6a0..edb996ca3d 100644 --- a/etc/test_lifecycle.yml +++ b/etc/test_lifecycle.yml @@ -1,4 +1,7 @@ # This file is used to tag JS tests that run under resmoke.py. # The content of this file lives in the https: //github.com/mongodb/mongo-test-metadata repository. # It is fetched automatically as part of the compile task. -selector: {} +selector: + js_test: + jstests/core/all.js: + - unreliable

      This issue was introduced by my changes from 046a5a0 as part of SERVER-31470 because resmoke.py exits with a non-zero return code if any tests failed (regardless of the evergreen_status it'll use in the report.json file).

            Assignee:
            backlog-server-dag [DO NOT ASSIGN] Backlog - Decision Automation Group (DAG) (Inactive)
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: