Consider an update that changes a document’s shard key, such that the document moves from one shard to another. Currently change streams show such updates as 2 separate events: delete and insert. Those events usually appear in that order but can appear as insert-then-delete.

TAR assumed, when designing mongosync’s embedded verifier, that the delete would always precede the insert, so we’re now having to implement a workaround. (cf. REP-4857)

Even if the server guaranteed delete-then-insert, though, this would still be suboptimal because, to a change stream reader, the document in question will appear to be “temporarily missing” during the time period—however short—between the delete and insert. For real-time applications this could cause durability problems.

In a Slack thread just now with ivan.fefer@mongodb.com and mickey.winters@mongodb.com, an alternate solution arose. If we can update the oplog format to indicate which insert & delete operations relate to shard-key updates, it may be possible instead to:

1. Omit the oplog’s delete from the change stream, and
2. Report the oplog’s insert as a replace event.

This way, change stream consumers only see a single replace event. This solves both the ordering and “temporarily missing” problems.

(NB: the oplog’s existing fromMigrate flag concerns chunk migrations, not shard-key updates, so it’s likely unhelpful here.)

cc max.hirschhorn@mongodb.com, michael.mcclimon@mongodb.com, craven.huynh@mongodb.com