We modified the way we handle read preference in mongosync:

• On the source, we respect whatever the user requested and use PrimaryPreferred if they did not set any preference.
• On the destination, we overwrite whatever the user requested with Primary.

If the user reverses the replication direction, we will apply the same logic on the new source and new destination.

e.g.

• User asks for nearest on source and secondaryPreferred on destination.
• We will use nearest on source (as requested) and Primary on destination (overwritten).
• If they call Reverse, we will use secondaryPreferred on the new source (as requested) and Primary on new destination (overwritten).

  Description of Linked Ticket

  The following scenario is possible:
  1. a. Replicator persists state on destination as COMMITTING
  1. b. State COMMITTING is saved on majority of nodes, but not some secondary S, which still thinks the state is RUNNING.
  2. Replicator is killed.
  3. Replicator restarts, and tries to restore its resume data. Since it uses PrimaryPreferred, if the primary is unavailable the replicator may read from S, and restore its state as RUNNING.

We may also see similar issues whenever we hit the api/v1/progress endpoint: it may return stale data.

To solve this, we should:
1. Force destination readPreference to be primary
2. Recreate mongo clients on reverse (since the destination should have the readPreference set)

Moved part on linearizability to https://jira.mongodb.org/browse/REP-854