The changes from a4d2c52 as part of
SERVER-28991 made it so a dedicated "flush thread" is used to do the actually sending of logs to logkeeper. The implementation was rewritten from using a custom AlarmClock thread for each logging handler to instead use a sched.scheduler instance to manage when to flush logs to logkeeper after every 10 seconds of the test running.
The change from AlarmClock.dismiss() to sched.scheduler.cancel() was flawed in that canceling the next flush event may fail if the BufferedHandler.flush() method is being concurrently called. This can be seen in how the BufferedHandler.close() method attempts to cancel self.__flush_event but ignores the return value.
As mathew.robinson and I saw when debugging together, it is therefore possible for a concurrent BufferedHandler.flush() call to schedule another flush event after sched.scheduler.cancel() fails to cancel the current one which leads to BufferedHandler.flush() being called perpetually.
A straightforward fix is to have BufferedHandler.close() record it was called and to have BufferedHandler.flush() not continuing to schedule itself again in that case.