[SERVER-40592] Uncaught exception in resmoke.py job thread due to logkeeper unavailability when tearing down fixture Created: 11/Apr/19  Updated: 29/Oct/23  Resolved: 03/May/19

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.11

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-42228 LoggerRuntimeConfigError exceptions c... Closed
related to SERVER-42622 resmoke.py doesn't attempt to tear th... Closed
is related to SERVER-28940 Make resmoke fixture setup/teardown t... Closed
is related to SERVER-35472 resmoke.py shouldn't fall back to std... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: STM 2019-05-20
Participants:
Linked BF Score: 5
Story Points: 2

 Description   

The call to teardown_fixture() should happen in a try/except block.

  • If there's a buildscripts.resmokelib.errors.LoggerRuntimeConfigError exception, then we should record the message with self.logger.error("%s", err) to the task logs.
  • If there's some other kind of exception, then we should record the message and Python stacktrace with self.logger.exception("Encountered an error when tearing down %s: %s", self.fixture, err) to the task logs. We should additionally set the teardown_flag.

[2019/04/08 21:07:45.443] Exception in thread Thread-1:
[2019/04/08 21:07:45.443] Traceback (most recent call last):
[2019/04/08 21:07:45.443]   File "/opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/python3-v3.lJe/lib/python3.7/threading.py", line 917, in _bootstrap_inner
[2019/04/08 21:07:45.443]     self.run()
[2019/04/08 21:07:45.443]   File "/opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/python3-v3.lJe/lib/python3.7/threading.py", line 865, in run
[2019/04/08 21:07:45.443]     self._target(*self._args, **self._kwargs)
[2019/04/08 21:07:45.443]   File "/data/mci/6e26eb78c635416bfda724f51d2fa812/src/buildscripts/resmokelib/testing/job.py", line 102, in __call__
[2019/04/08 21:07:45.443]     if teardown_flag is not None and not self.teardown_fixture():
[2019/04/08 21:07:45.443]   File "/data/mci/6e26eb78c635416bfda724f51d2fa812/src/buildscripts/resmokelib/testing/job.py", line 64, in teardown_fixture
[2019/04/08 21:07:45.443]     test_case(self.report)
[2019/04/08 21:07:45.443]   File "/opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/python3-v3.lJe/lib/python3.7/unittest/case.py", line 663, in __call__
[2019/04/08 21:07:45.443]     return self.run(*args, **kwds)
[2019/04/08 21:07:45.443]   File "/opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/python3-v3.lJe/lib/python3.7/unittest/case.py", line 588, in run
[2019/04/08 21:07:45.443]     result.startTest(self)
[2019/04/08 21:07:45.443]   File "/data/mci/6e26eb78c635416bfda724f51d2fa812/src/buildscripts/resmokelib/testing/report.py", line 112, in startTest
[2019/04/08 21:07:45.443]     test.logger)
[2019/04/08 21:07:45.443]   File "/data/mci/6e26eb78c635416bfda724f51d2fa812/src/buildscripts/resmokelib/logging/loggers.py", line 205, in new_test_logger
[2019/04/08 21:07:45.443]     " test_id".format(test_basename))
[2019/04/08 21:07:45.443] buildscripts.resmokelib.errors.LoggerRuntimeConfigError: Encountered an error configuring buildlogger for test job0_fixture_teardown: Failed to get a new test_id



 Comments   
Comment by Githook User [ 03/May/19 ]

Author:

{'email': 'max.hirschhorn@mongodb.com', 'name': 'Max Hirschhorn', 'username': 'visemet'}

Message: SERVER-40592 Handle exceptions from

{setup,teardown}

_fixture().
Branch: master
https://github.com/mongodb/mongo/commit/21e227e2c6a82bc9405cc36cf8779c1fe0723ea2

Comment by Max Hirschhorn [ 02/May/19 ]

A similar case also impacts setup_fixture(). For example: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_aggregation_8721322fedc70435bb1e6c2fc7174ce021433e6f_19_05_02_17_10_01

Generated at Thu Feb 08 04:55:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.