|
Takeaways from further investigation on the impacts of this behavior:
- If mongos starts up without keys, it will not include cluster time metadata in any response (i.e. $clusterTime or operationTime) until it asynchronously discovers keys through a background thread. The thread refreshes every 200ms w/ linear backoff until keys are found w/ a max interval of 10 minutes, so in normal operation this period should be short. I'm not positive how a driver handles responses without clusterTime metadata, but given this has been the behavior in master for some time, it doesn't seem to have broken any driver tests.
- With auth on, I don't believe conforming drivers shouldn't experience any causal consistency violations, because either the driver has not received valid clusterTime metadata and cannot perform afterClusterTime reads, or it has received metadata from another mongos, which will be rejected by the mongos w/o keys because it cannot validate the proof included with $clusterTime.
- With auth off, however, I think it is possible to violate causal consistency because mongos does not return clusterTime metadata without keys, but skips validating proofs with received clusterTimes. So a mongos that has received valid clusterTimes at some point may perform a write against the mongos w/o keys, receive no new clusterTimes, then perform a read using the stale metadata, which will not reflect the earlier write.
- Without waiting for keys, it's unlikely we'll discover bugs with key generation / rotation, because of the "mongosWaitsForKeys" parameter in ShardingTest, which was added before it was decided mongos should wait for keys at startup and masked this bug by waiting for mongos to return keys in the ShardingTest constructor. If we do this ticket, I'd recommend removing that parameter as well.
|