-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Fully Compatible
-
ALL
-
-
Query F (02/01/16)
-
0
The resmokelib.logging.handlers.BufferedHandler uses a timer thread to periodically send a POST request with logs to logkeeper. A separate thread to flush() and close() the logging handlers (thereby joining the timer thread) was created in order to avoid exhausting the number of threads that can be created on a 32-bit system.
Due to a bug in the processing of its _LOGGER_QUEUE, the flush thread may try to call flush() on a logging handler after the interpreter has been marked as uninitialized and the module dictionary has been destroyed. This tends to manifest as either
- "Fatal Python error: PyImport_GetModuleDict: no module dictionary!"
- "Exception in thread FlushThread (most likely raised during interpreter shutdown)"
Extracted the relevant bits from pythonrun.c below. The FlushThread is a daemon thread in order to avoid requiring a user to wait for all the output to be logged after triggering an interrupt. This means that the FlushThread is still running after wait_for_thread_shutdown() has been called and it will attempt to send POST requests to logkeeper with any remaining logs.
/* Wait until threading._shutdown completes, provided the threading module was imported in the first place. The shutdown routine will wait until all non-daemon "threading" threads have completed. */ static void wait_for_thread_shutdown(void) { #ifdef WITH_THREAD PyObject *result; PyThreadState *tstate = PyThreadState_GET(); PyObject *threading = PyMapping_GetItemString(tstate->interp->modules, "threading"); if (threading == NULL) { /* threading not imported */ PyErr_Clear(); return; } result = PyObject_CallMethod(threading, "_shutdown", ""); if (result == NULL) PyErr_WriteUnraisable(threading); else Py_DECREF(result); Py_DECREF(threading); #endif } void Py_Finalize(void) { PyInterpreterState *interp; PyThreadState *tstate; if (!initialized) return; wait_for_thread_shutdown(); /* The interpreter is still entirely intact at this point, and the * exit funcs may be relying on that. In particular, if some thread * or exit func is still waiting to do an import, the import machinery * expects Py_IsInitialized() to return true. So don't say the * interpreter is uninitialized until after the exit funcs have run. * Note that Threading.py uses an exit func to do a join on all the * threads created thru it, so this also protects pending imports in * the threads created via Threading. */ call_sys_exitfunc(); initialized = 0; ... /* Destroy all modules */ PyImport_Cleanup(); ... }