Our primary server ran out of disk space (due to us turning on verbose logging to diagnose a different problem). This then, correctly, caused the primary to step down. However, it was a single operation ahead of any of the secondaries.

We then saw that none of the secondaries would take over as primary because the primary would veto (due to being ahead). And so, our set was left without a primary until we killed the mongo process on the primary (freeing up space by deleting the large log files had no affect as mongo still had a lock on the log file).

In this situation it would be good if:

• the primary had carried on replicating - allowing the secondaries to catch up and take over
• the primary had completely died - we can deal with the rollback issue and write safety would have meant we were ok here anyway