I believe this is behavior is "as designed" with how we only call flush() on the log file and not fsync(). This is undesirable because the mongod logs are a record of operations that happened against the database which is useful to consult when investigating a data inconsistency failure. After discussing with milkie via Slack, I think there a few ways to address this issue:
1. Change powertest.py to send the mongod process a SIGUSR1 signal to cause it to reopen its log file (the mongod process is started with --logappend). I don't think this has the intended effect because close() doesn't implicitly call fsync().
2. Change powertest.py to rsync the mongod.log file back to the controller before crashing the machine. This seems potentially tricky with how we'd like for the machine to be crashed immediately after the journaled canary insert finishes.
3. Add a testing-only mode where mongod opens the log file with O_DSYNC. (http://man7.org/linux/man-pages/man2/open.2.html is a handy reference.)
4. Add a testing-only command that causes mongod to fsync() its log file.