During the course of SERVER-55501, we added an optimization for oplog diff application for certain scenarios where we know about the structure of the pre-image and the diff, and can guarantee that fields which are inserted by the diff do not already exist in the pre-image.

In the case of updates that happen as a result of timeseries inserts through the normal BucketCatalog machinery, we know that the resulting oplog entry which is applied on the primary should satisfy these conditions. Additionally, we know that the corresponding entry when applied on a secondary in steady state should also qualify.

What we found is that tenant migrations throw some wrenches in the work here. In particular, it looks like we need to disable the optimization on the primary even when the write goes through the bucket catalog, if the write comes from a tenant migration replaying the oplog. After talking it through a bit, lingzhi.deng and dan.larkin-york came to the conclusion that the secondary should in theory be able to apply any entries generated from the primary blindly with the optimization, without checking if they resulted from a tenant migration - however, this didn't appear to be the case. Some still resulted in field duplication, and thus required the check for tenant migration source.

It remains unclear why we sometimes generate these entries which require the strict idempotency guarantees which normally are not required for writes coming through the BucketCatalog. It may be that something is going wrong at the BucketCatalog layer, or it may be that tenant migrations are doing something unexpected, or any number of other things. The goal of this ticket is simply to understand what's going on here.