[SERVER-22142] resmoke.py's FlushThread attempts to reference imported members during Python interpreter shutdown Created: 12/Jan/16 Updated: 18/Nov/16 Resolved: 15/Jan/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.3, 3.3.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Max Hirschhorn |
| Resolution: | Done | Votes: | 0 |
| Labels: | test-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||
| Backport Completed: | |||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | This seemed to fail most often on Windows (Python 2.7.3) when the Python interpreter was relatively short-lived (~30 seconds). To achieve that, the failpoints and audit tasks were run repeatedly using the following patch:
The following patch appears to be sufficient to reproduce on a Windows spawn host.
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Query F (02/01/16) | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||||||||||||||||||||||
| Description |
|
The resmokelib.logging.handlers.BufferedHandler uses a timer thread to periodically send a POST request with logs to logkeeper. A separate thread to flush() and close() the logging handlers (thereby joining the timer thread) was created in order to avoid exhausting the number of threads that can be created on a 32-bit system. Due to a bug in the processing of its _LOGGER_QUEUE, the flush thread may try to call flush() on a logging handler after the interpreter has been marked as uninitialized and the module dictionary has been destroyed. This tends to manifest as either
Extracted the relevant bits from pythonrun.c below. The FlushThread is a daemon thread in order to avoid requiring a user to wait for all the output to be logged after triggering an interrupt. This means that the FlushThread is still running after wait_for_thread_shutdown() has been called and it will attempt to send POST requests to logkeeper with any remaining logs.
|
| Comments |
| Comment by Githook User [ 27/Jan/16 ] |
|
Author: {u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}Message: This prevents the FlushThread from running while the interpreter is Also fixed an issue where a KeyboardInterrupt while running multiple (cherry picked from commit 00c8c64f6cd9b0d236d50a8a1cf7152e0367a5cd) |
| Comment by Githook User [ 15/Jan/16 ] |
|
Author: {u'username': u'visemet', u'name': u'Max Hirschhorn', u'email': u'max.hirschhorn@mongodb.com'}Message: This prevents the FlushThread from running while the interpreter is Also fixed an issue where a KeyboardInterrupt while running multiple |