Documentation Request Summary:

Upgrade downgrade docs need to mention that after changing the featureCompatibilityVersion mongos needs to be bounced to pick up the changes in the causal consistency behavior. It appears that normal procedure will do just that but its better to stress in the case users decide to run the command.

Engineering Ticket Description:

Currently mongos does not return operationTime until the signing keys become available. This prevents sessions to be causally consistent as the keys are not available instantaneously.

Note:
returning just operationTime even if there are no keys as it does not require keys introduces issues related to a multi-shard execution without gossiping logical time. Its ok for a single RS as it does not need to gossip the clusterTime as its always matches the oplog time

Suggested implementation will block in mongos initialization until keys are available if its accessing a sharded cluster

1.
in initializeSharding: https://github.com/mongodb/mongo/blob/r3.5.10/src/mongo/s/server.cpp#L190
add a blocking call until keys are available:

status = waitForSigningKeys(opCtx);
    if (!status.isOK()) {
        return status;
    }

2.
implement waitForSigningKeys as (very similar to waitForShardRegistryReload)

Status waitForSigningKeys(OperationContext* opCtx) {
    while (true) {
        auto stopStatus = opCtx->checkForInterruptNoAssert();
        if (!stopStatus.isOK()) {
            return stopStatus;
        }

        try {
           auto logicalTimeValidator = LogicalTimeValidator::get(opCtx)
           
            if (logicalTimeValidator->shouldGossipLogicalTime()) {
                return Status::OK();
            }
            sleepFor(kRetryInterval);
            continue;
        } catch (const DBException& ex) {
            Status status = ex.toStatus();
            warning()
                << "Error waiting for signing keys, sleeping for 2 seconds and trying again"
                << causedBy(status);
            sleepFor(kRetryInterval);
            continue;
        }
    }
}