Description

For mongod version < 4.0, if we have below upgrade/downgrade sequence.

1) Start a replica set in pv1.

2) Downgrade to pv0.

3) Before the secondaries downgrade to pv0, the user upgrades to pv1 again. 

Then it can lead to failures described in SERVER-38505, SERVER-38366 and SERVER-38504.

In order to prevent those failures from occurring, we should add a comment in modify-replica-set-protocol-version stating that "Before user upgrades or downgrades the replica set protocol version, the user should make sure that at least one oplog entry generated from the current protocol version has replicated and applied to all secondaries. And, we can get the node's last applied oplog entry optime information from  either replSetGetStatus.optimes.appliedOpTime or getLastError.lastOp. If the current protocol version is pv0, then replSetGetStatus.optimes.appliedOpTime & getLastError.lastOp will have an optime with term field ("t") as -1. If it is pv1, then the term field will have a value greater than -1".

Scope

Update Modify Replica Set Protocol Version to call out that users must wait until the protocol version change has replicated and applied to all members of the replica set. Provide logic for testing as above.

Possibly generally dissuade users from rapidly changing pv values.

Note: 3.6 deprecates pv0. Apply to 3.6 and backport to 3.2 (when pv1 was introduced)