Details
-
Bug
-
Resolution: Done
-
Major - P3
-
None
-
Fully Compatible
-
ALL
-
-
Query F (02/01/16)
-
0
Description
The resmokelib.logging.handlers.BufferedHandler uses a timer thread to periodically send a POST request with logs to logkeeper. A separate thread to flush() and close() the logging handlers (thereby joining the timer thread) was created in order to avoid exhausting the number of threads that can be created on a 32-bit system.
Due to a bug in the processing of its _LOGGER_QUEUE, the flush thread may try to call flush() on a logging handler after the interpreter has been marked as uninitialized and the module dictionary has been destroyed. This tends to manifest as either
- "Fatal Python error: PyImport_GetModuleDict: no module dictionary!"
- "Exception in thread FlushThread (most likely raised during interpreter shutdown)"
Extracted the relevant bits from pythonrun.c below. The FlushThread is a daemon thread in order to avoid requiring a user to wait for all the output to be logged after triggering an interrupt. This means that the FlushThread is still running after wait_for_thread_shutdown() has been called and it will attempt to send POST requests to logkeeper with any remaining logs.
/* Wait until threading._shutdown completes, provided |
the threading module was imported in the first place. |
The shutdown routine will wait until all non-daemon |
"threading" threads have completed. */ |
static void
|
wait_for_thread_shutdown(void)
|
{
|
#ifdef WITH_THREAD
|
PyObject *result; |
PyThreadState *tstate = PyThreadState_GET(); |
PyObject *threading = PyMapping_GetItemString(tstate->interp->modules, |
"threading"); |
if (threading == NULL) { |
/* threading not imported */ |
PyErr_Clear();
|
return; |
}
|
result = PyObject_CallMethod(threading, "_shutdown", ""); |
if (result == NULL) |
PyErr_WriteUnraisable(threading);
|
else |
Py_DECREF(result);
|
Py_DECREF(threading);
|
#endif
|
}
|
|
void
|
Py_Finalize(void)
|
{
|
PyInterpreterState *interp; |
PyThreadState *tstate; |
|
if (!initialized) |
return; |
|
wait_for_thread_shutdown();
|
|
/* The interpreter is still entirely intact at this point, and the |
* exit funcs may be relying on that. In particular, if some thread |
* or exit func is still waiting to do an import, the import machinery |
* expects Py_IsInitialized() to return true. So don't say the |
* interpreter is uninitialized until after the exit funcs have run. |
* Note that Threading.py uses an exit func to do a join on all the |
* threads created thru it, so this also protects pending imports in |
* the threads created via Threading. |
*/ |
call_sys_exitfunc();
|
initialized = 0; |
|
...
|
|
/* Destroy all modules */ |
PyImport_Cleanup();
|
|
...
|
}
|