-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Cluster Scalability
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Currently for jumbo chunks, the chunk migration donor gets the records to transfer to the recipient in batches by scanning the live shard key index. Reading the live index has an issue where an update moves a record later in the index and is encountered again and transferred again. This then causes duplicate key errors which prevents migration from succeeding (example: HELP-86419).
To improve on this, we can consider doing the index scan at a stable snapshot time. This then removes the issue where a concurrent update causes a record to be encountered twice.
The challenge is getting a correct snapshot time to read at and handling what happens if the snapshot history is gone.
- For getting a good snapshot time: one idea is when starting the scan, we use/store lastApplied (last applied oplog time) and wait for that to be majority committed. This should work because the scan happens after the migration op-observers used for transferring write changes are set-up. So any changes after the snapshot read will be transferred to the recipient.
- This time does not need to be persisted, because migration does not survive failovers.
- For handling snapshot history missing, we can fail migration and have it retry.
As a backup we should still allow the old behavior without snapshot reads in case some customer doesn't store snapshot history.