Refining a collection's shard key pattern modifies the sharding metadata stored on the config server without triggering any physical data movement.

Let's say the collection's shard key pattern is refined from {x: 1} to {x: 1, y: 1}. The observation is when the chunk bounds were expressed only in terms of 'x' values, for a particular value of 'x', all the possible values of 'y' lived on that one shard. So padding the chunk bounds in the sharding metadata when the collection's shard key pattern is refined expresses that equivalency and allows the balancer + auto-splitter to later split the ranges into smaller ones and migrate them.

However, the ShardKeyPattern::isExtendedBy() function called by the _configsvrRefineCollectionShardKey command only checks whether the field names in the new shard key pattern have the current shard key pattern as a prefix. In particular, it doesn't validate that the value for each of the field names (i.e. 1 or "hashed") is the same between the two shard key patterns. This unintentionally permits the collection's shard key pattern {x: 1} to be "refined" to {x: "hashed"} when there's a supporting index. However, the existing data is distributed using {x: 1} and not {x: "hashed"} so some of the existing data will become inaccessible. The existing data is still present on all of the shards and can be queried when using read concern level "available" but likely unowned by the shard and therefore cannot be updated.

// Validate the given shard key (i) extends the current shard key, (ii) has a "useful"
// index, and (iii) the index in question has no null entries.
uassert(ErrorCodes::InvalidOptions,
        str::stream() << "refineCollectionShardKey shard key "
                      << newShardKeyPattern.toString()
                      << " does not extend the current shard key "
                      << oldShardKeyPattern.toString(),
        oldShardKeyPattern.isExtendedBy(newShardKeyPattern));

We should change ShardKeyPattern::isExtendedBy() is perform this additional validation to prevent users from getting into a bad state.