Add a class that can run a BackgroundJob on the config servers to initialize sharding awareness on any shards in config.shards that are not marked as sharding aware (as represented by a flag field in config.shards).
The BackgroundJob can be kicked off on transition to primary or by an OpObserver watching for inserts to config.shards.
While the cluster upgrade is in progress:
- If a non-3.4 shard is added by an old mongos OR the new _configsvrAddShard process, the shardIdentity document will be inserted into the shard and remain latent until the shard is upgraded to 3.4 and restarted with --shardsvr, at which point the shard will become sharding aware. The configs will incorrectly but harmlessly believe the shard is sharding aware (that is, the shard will be marked as "sharding aware" in config.shards). This is because the insert of the shardIdentity document won't have been impeded by an OpObserver on the shard checking for --shardsvr.
- If a 3.4 shard is added through the new _configsvrAddShard process, all works as designed for a 3.4 cluster.
- If a 3.4 shard started with --shardsvr is added by an old mongos, the OpObserver on the configs will asynchronously initialize sharding awareness on it by kicking off the BackgroundJob.
- If a 3.4 mongod without --shardsvr is added by an old mongos, the OpObserver will kick off the asynchronous BackgroundJob, but the mongod will reject the shardIdentity document. The old mongos will believe the shard has been added, but all sharding requests the mongos sends to it will be rejected by the shard. TODO: Should the BackgroundJob remove the shard's entry from config.shards, so that the old mongos stops trying to communicate with it as soon as it reloads the ShardRegistry?