[SERVER-34420] Set the idle event in the stepdown thread even if the thread exits in stepdown.py Created: 11/Apr/18  Updated: 29/Oct/23  Resolved: 04/May/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.0.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Samyukta Lanka Assignee: David Bradford (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Sprint: TIG 2018-05-21
Participants:
Linked BF Score: 0
Story Points: 2

 Description   

In stepdown.py, if the stepdown thread exits before setting the idle event, we endlessly keep waiting to pause the stepdown thread.

[2018/04/11 00:18:17.596] Thread 140674028517120:
[2018/04/11 00:18:17.596]   File "/opt/mongodbtoolchain/v2/lib/python2.7/threading.py", line 774, in __bootstrap
[2018/04/11 00:18:17.596]     self.__bootstrap_inner()
[2018/04/11 00:18:17.596]   File "/opt/mongodbtoolchain/v2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
[2018/04/11 00:18:17.597]     self.run()
[2018/04/11 00:18:17.597]   File "/opt/mongodbtoolchain/v2/lib/python2.7/threading.py", line 754, in run
[2018/04/11 00:18:17.597]     self.__target(*self.__args, **self.__kwargs)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/job.py", line 45, in __call__
[2018/04/11 00:18:17.597]     self._run(queue, interrupt_flag)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/job.py", line 83, in _run
[2018/04/11 00:18:17.597]     self._execute_test(test)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/job.py", line 115, in _execute_test
[2018/04/11 00:18:17.597]     self._run_hooks_after_tests(test)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/job.py", line 168, in _run_hooks_after_tests
[2018/04/11 00:18:17.597]     self._run_hook(hook, hook.after_test, test)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/job.py", line 121, in _run_hook
[2018/04/11 00:18:17.597]     hook_function(test, self.report)
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/hooks/stepdown.py", line 75, in after_test
[2018/04/11 00:18:17.597]     self._stepdown_thread.pause()
[2018/04/11 00:18:17.597]   File "/data/mci/bfe6b97b5e63a6c8969ac499179caac2/src/buildscripts/resmokelib/testing/hooks/stepdown.py", line 157, in pause
[2018/04/11 00:18:17.597]     self._is_idle_evt.wait()
[2018/04/11 00:18:17.597]   File "/opt/mongodbtoolchain/v2/lib/python2.7/threading.py", line 614, in wait
[2018/04/11 00:18:17.597]     self.__cond.wait(timeout)
[2018/04/11 00:18:17.597]   File "/opt/mongodbtoolchain/v2/lib/python2.7/threading.py", line 340, in wait
[2018/04/11 00:18:17.597]     waiter.acquire()

This could be fixed in stepdown.py:

def _step_down_all(self):
    self._is_idle_evt.clear()
    try:
        for rs_fixture in self._rs_fixtures:
            self._step_down(rs_fixture)
    finally:
        self._is_idle_evt.set()



 Comments   
Comment by Githook User [ 04/May/18 ]

Author:

{'email': 'david.bradford@mongodb.com', 'name': 'David Bradford', 'username': 'dbradf'}

Message: SERVER-34420: Set idle event in stepdown thread even if there is a failure
Branch: master
https://github.com/mongodb/mongo/commit/fabcd8eb1618a870fa5597d296505b03779de105

Generated at Thu Feb 08 04:36:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.