During rollback as we scan backwards in our oplog finding entries that must be rolled back, we save in memory copies of the the entire oplog entry here and here, if I'm reading the code correctly. If there is a lot of oplog to roll back this could be a lot of memory, resulting in OOM.

This code appears to have completely changed in 3.6, and I think it may not have the same problem. If so we can mark this issue fixed in 3.6 and/or dup it to the ticket that fixed the issue, using this ticket to track the issue and the fix.