[SERVER-22165] Deadlock in resmoke.py between logger pipe and timer thread Created: 13/Jan/16 Updated: 18/Nov/16 Resolved: 19/Jan/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.3, 3.3.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Jonathan Abrahams |
| Resolution: | Done | Votes: | 0 |
| Labels: | test-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Operating System: | ALL | ||||
| Backport Completed: | |||||
| Sprint: | TIG F (01/29/16) | ||||
| Participants: | |||||
| Linked BF Score: | 0 | ||||
| Description |
|
The logger pipe will acquire the lock for guarding the handler's buffer, and then will attempt to acquire the lock for guarding the timer's state in order to snooze the timer. The timer thread will call func while holding the lock on its state, which will attempt to acquire the lock for guarding the handler's buffer by calling flush() on the handler. This can lead to a deadlock between the two threads. When this situations occurs, resmoke.py will no longer be able to consume output from the stdout/stderr pipe of the subprocess (e.g. the mongo shell). The pipe will eventually fill up and subsequent writes to it will cause that thread to block. The test's operation will then stall and likely cause the task in Evergreen to time out.
Proposal is to modify resmokelib.utils.timer.AlarmClock.run() to release the lock on the timer's state before calling func, and then reacquire it in order to set self.snoozed = False. This prevents the timer thread from causing an invalid lock ordering so long as func is safe to call on its own. |
| Comments |
| Comment by Githook User [ 19/Jan/16 ] |
|
Author: {u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}Message: (cherry picked from commit 8aa1202a37900752c102ce7b324f41853ec5c4c9) |
| Comment by Githook User [ 19/Jan/16 ] |
|
Author: {u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}Message: |