Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-87187

Shell crashes at the end of a jstest with embedded router and stepdowns

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • Catalog and Routing
    • ALL
    • CAR Team 2024-03-18, CAR Team 2024-04-01, CAR Team 2024-04-15, CAR Team 2024-04-29

      Sample test to reproduce the issue. An example log is also attached. The test itself completed successfully, so it looks that the crash happens during shutdown.

      [js_test:test_ftdc_new_format] ShardingTest stopped all shards, took 23932ms for 1 shards.
      [js_test:test_ftdc_new_format] *** ShardingTest test_ftdc_new_format completed successfully in 49.804 seconds ***
      
      [...]
      
      [executor:js_test:job0] 14:29:47.022Z test_ftdc_new_format.js ran in 52.63 seconds: failed.
      [executor:js_test:job0] 14:29:47.024Z Encountered an error during test execution.
      Traceback (most recent call last):
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 194, in _run_single_copy
          test_case.run_test()
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 160, in run_test
          self._execute(proc)
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 181, in _execute
          raise self.failureException("%s failed" % (self.short_description()))
      AssertionError: JSTest jstests/sharding/test_ftdc_new_format.js failed
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/job.py", line 115, in __call__
          self._run(queue, interrupt_flag, teardown_flag, hook_failure_flag)
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/job.py", line 163, in _run
          self._execute_test(test, hook_failure_flag)
        File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CBC/lib/python3.10/contextlib.py", line 79, in inner
          return func(*args, **kwds)
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/job.py", line 219, in _execute_test
          raise test.propagate_error
        File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CBC/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
          yield
        File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CBC/lib/python3.10/unittest/case.py", line 591, in run
          self._callTestMethod(testMethod)
        File "/opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/python3-v4.CBC/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
          method()
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 244, in run_test
          self._run_single_copy()
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 198, in _run_single_copy
          self._raise_if_unsafe_exit(self.return_code)
        File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 262, in _raise_if_unsafe_exit
          raise self.propagate_error  # pylint: disable=raising-bad-type
      buildscripts.resmokelib.errors.UnsafeExitError: Mongo shell exited with code -6 while running jstest test_ftdc_new_format.js. Further test execution may be unsafe.
      [executor] 14:29:47.024Z Waiting for threads to complete
      [executor:js_test:job0] 14:29:47.024Z Running job0_fixture_teardown...
      [js_test:job0_fixture_teardown] Starting the teardown of NoOpFixture (Job #0).
      [js_test:job0_fixture_teardown] Finished the teardown of NoOpFixture (Job #0).
      [js_test:job0_fixture_teardown] Not running in Evergreen, skipping symbolization
      [executor:js_test:job0] 14:29:47.025Z job0_fixture_teardown ran in 0.00 seconds: no failures detected.
      [executor] 14:29:47.025Z Threads are completed!
      [executor] 14:29:47.025Z Summary of latest execution: 3 test(s) ran in 52.65 seconds (2 succeeded, 0 were skipped, 0 failed, 1 errored)
          The following tests had errors:
              jstests/sharding/test_ftdc_new_format.js
                  Traceback (most recent call last):
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 194, in _run_single_copy
                      test_case.run_test()
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 160, in run_test
                      self._execute(proc)
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 181, in _execute
                      raise self.failureException("%s failed" % (self.short_description()))
                  AssertionError: JSTest jstests/sharding/test_ftdc_new_format.js failed
                  
                  During handling of the above exception, another exception occurred:
                  
                  Traceback (most recent call last):
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 244, in run_test
                      self._run_single_copy()
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 198, in _run_single_copy
                      self._raise_if_unsafe_exit(self.return_code)
                    File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 262, in _raise_if_unsafe_exit
                      raise self.propagate_error  # pylint: disable=raising-bad-type
                  buildscripts.resmokelib.errors.UnsafeExitError: Mongo shell exited with code -6 while running jstest test_ftdc_new_format.js. Further test execution may be unsafe.
                  
          If you're unsure where to begin investigating these errors, consider looking at tests in the following order:
              jstests/sharding/test_ftdc_new_format.js
      [resmoke] 14:29:47.026Z ================================================================================
      [resmoke] 14:29:47.026Z Summary of sharding_csrs_continuous_config_stepdown suite: 3 test(s) ran in 52.65 seconds (2 succeeded, 0 were skipped, 0 failed, 1 errored)
      The following tests had errors:
          jstests/sharding/test_ftdc_new_format.js
              Traceback (most recent call last):
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 194, in _run_single_copy
                  test_case.run_test()
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 160, in run_test
                  self._execute(proc)
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/interface.py", line 181, in _execute
                  raise self.failureException("%s failed" % (self.short_description()))
              AssertionError: JSTest jstests/sharding/test_ftdc_new_format.js failed
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 244, in run_test
                  self._run_single_copy()
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 198, in _run_single_copy
                  self._raise_if_unsafe_exit(self.return_code)
                File "/home/ubuntu/git/mongo-worktrees/scratch/buildscripts/resmokelib/testing/testcases/jstest.py", line 262, in _raise_if_unsafe_exit
                  raise self.propagate_error  # pylint: disable=raising-bad-type
              buildscripts.resmokelib.errors.UnsafeExitError: Mongo shell exited with code -6 while running jstest test_ftdc_new_format.js. Further test execution may be unsafe.
              
      If you're unsure where to begin investigating these errors, consider looking at tests in the following order:
          jstests/sharding/test_ftdc_new_format.js
      [resmoke] 14:29:47.026Z Exiting with code: 1
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            aitor.esteve@mongodb.com Aitor Esteve Alvarado
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: