Because the readPreference is PrimaryPreferred, succh commands enter this branch in DBClientRS::query(), which

1) selects a host within the replica set based on the readPreference

2) runs the query against that host

3) catches any exception and marks the selected host as failed

4) retries against a different host

If mongos is stale, it targets an incorrect set of shards. Within each shard, it first targets the primary node (since the readPref is PrimaryPreferred). One of these nodes (that has donated a chunk since mongos refreshed) will return a stale config error.

The bug is that mongos does not bubble up the stale config error so that its routing table cache can be invalidated and the request re-routed. Instead, it catches the exception in a blanket catch statement low in the networking code, marks the shard's primary as failed, and retries from step 1 above. On the retry, mongos selects one of the shard's secondaries (since the readPref is PrimaryPreferred). Secondaries do not participate in shard versioning in 3.4 (this was fixed as the "Safe Secondary Reads" feature in 3.6 (PM-256)), so the selected host returns whatever data it has.

Users can work around this issue by using readPreference PrimaryOnly on 3.4 or upgrading to 3.6 and setting the feature compatibility version to 3.6.

Note that the only known command that manifests this issue is aggregate, and only when the collection goes from unsharded to sharded (because aggregate's sharded code path does not use the buggy DBClientRS::query()).