[SERVER-76779] moveChunk should wait until in-progress index builds are finished on recipient shard Created: 03/May/23  Updated: 29/Oct/23  Resolved: 22/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0, 7.0.0-rc6

Type: Bug Priority: Major - P3
Reporter: Pol Pinol Assignee: Pol Pinol
Resolution: Fixed Votes: 0
Labels: sharding-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0
Sprint: Sharding EMEA 2023-05-15, Sharding EMEA 2023-05-29, Sharding EMEA 2023-06-12, Sharding EMEA 2023-06-26
Participants:
Linked BF Score: 14

 Description   

When there is a moveChunk from one shard to another, in the migration_destination_manager.cpp (cloning indexes phase), the recipient shard is not waiting for in-progress index builds to be finished.

More in particular, the logic of filtering the existing indexes on the destination shard, is using removeExistingIndexesNoChecks that filters out read and in-progress indexes.

If the recipient shard assumes that in-progress indexes are already created indexes, we could potentially end up in a situation where the index is not finally created in that shard. There is no guarantee that in-progress index build to succeed.

The goal of this ticket is to fix that behaviour. In order to make that, we could make moveChunk wait for in-progress index build to be finished on the recipient shard.



 Comments   
Comment by Githook User [ 22/Jun/23 ]

Author:

{'name': 'Pol Pinol Castuera', 'email': 'pol.pinol@mongodb.com', 'username': 'PolPinol'}

Message: SERVER-76779 moveChunk should wait until in-progress index builds are finished on recipient shard

(cherry picked from commit 09884999cd6ed51aacf98c9b59851f225d1b8968)
Branch: v7.0
https://github.com/mongodb/mongo/commit/8a6a2d094b01b10b735972c894bec9d3397ab16e

Comment by Githook User [ 21/Jun/23 ]

Author:

{'name': 'Pol Pinol Castuera', 'email': 'pol.pinol@mongodb.com', 'username': 'PolPinol'}

Message: SERVER-76779 moveChunk should wait until in-progress index builds are finished on recipient shard
Branch: master
https://github.com/mongodb/mongo/commit/09884999cd6ed51aacf98c9b59851f225d1b8968

Comment by Max Hirschhorn [ 03/May/23 ]

There is no guarantee for in-progress index builds to succeed in being built on the donor shard. It is always possible for the index build to hit a duplicate key error or some other form of index key constraint. Having the recipient shard copy over in-progress index builds can still lead to index inconsistencies between shards in the other direction. Speculating the index build will end up succeeding may be preferable from a user perspective because probably the most likely outcome for the index build. However I do feel like the ultimate fix for shards having consistent indexes must be in the createIndexes command to have explicit orchestration for index builds across all shards.

Also note that refineCollectionShardKey followed by chunk migration leading to a missing shard key index has other impacts tracked in SERVER-47025.

Generated at Thu Feb 08 06:33:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.