In a recent help ticket we saw an occurrence of SERVER-118902 triggered by a priority takeover.

SERVER-118902 is a known bug where nodes can accept a write for term T after voting in term T+1 but before it selects a sync source that knows about T+1. If these writes need to be rolled back, then it can result in a scenario where a majority of nodes are in the ROLLBACK state, leading to a loss of write availability.

Step-down and failover make this unlikely or impossible by ensuring we do not do writes that will get rolled back. In step-down, the old primary ensures there is a candidate new primary and stops accepting writes that will conflict. In failover, the old primary is down, and so conflicting writes are less likely. [1]

In this incident the election was the result of priority takeover. It appears this did not trigger a graceful step-down, which resulted in 2 out of 3 nodes being in the ROLLBACK state for ~5 minutes.

To compound the impact, this happened as a side effect of vertical scaling (the preferred primary was taken down to change the instance type), which means from the customer POV the scaling mechanism hurt availability when it should have helped.

Closing criteria:

• Priority takeover does not risk putting nodes that vote in the election into ROLLBACK. (This could either mean solving SERVER-118902, or changing the priority takeover mechanism to use graceful step-down.)
• A test confirms that a priority takeover has a meaningful barrier between writes for terms T and T+1.

[1] Network partitions could potentially surface this bug, but I'm hand-waving over that.