[SERVER-22503] movePrimary builds indexes in foreground on destination shard Created: 07/Feb/16  Updated: 06/Dec/22  Resolved: 09/Dec/19

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.0.9, 3.2.1
Fix Version/s: 4.2.0

Type: Bug Priority: Major - P3
Reporter: James Wahlin Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Done Votes: 2
Labels: pm-1051-legacy-tickets
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to DOCS-7112 Add a warning that movePrimary may bu... Closed
related to DOCS-9366 Add a warning that copydb/copyDatabas... Closed
Assigned Teams:
Sharding
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Case:

 Description   

The movePrimary command is used to move primary database and unsharded collections from one shard to another. When movePrimary clones a collection from the donor to the recipient shard, it will create indexes using a MultiIndexBlock. It does not call MultiIndexBlock::allowBackgroundBuilding() before executing. Without this call all index builds will be performed in the foreground, regardless of whether {background: true} is set in a given index's specification.

Performing foreground index builds on the recipient shard will block all database read/write activity for the duration of each index build. This can be a significant period of time depending on data size and hardware resources available.

We should look to build indexes in the background during movePrimary. If not possible, we should at a minimum update the movePrimary documentation to reflect.

This also impacts CloudManager shard removal via automation, in the case where the shard being removed is the primary database for unsharded collections.

For movePrimary index build code see:
https://github.com/mongodb/mongo/blob/r3.2.1/src/mongo/db/cloner.cpp#L358-L370



 Comments   
Comment by Sheeri Cabral (Inactive) [ 09/Dec/19 ]

fixed in 4.1.8. Hybrid Index builds project fixed the negative impact.

Comment by Eric Milkie [ 14/Jun/18 ]

It turns out that movePrimary clones all the data and then builds indexes, similar to initial sync. So one way we could fix this is by building the indexes first and then cloning the data.
I have also linked this ticket to the Hybrid Index Build project, which will not lock the db while the index build is progressing.

Comment by Eric Milkie [ 22/May/18 ]

Putting this ticket back in Needs Triage, as I don't believe this is a problem, at least in 4.0. I would expect the movePrimary command to build indexes on empty collections, not ones with data in them.
I also removed this ticket from the Simultaneous Indexing on All Nodes epic.

Comment by Spencer Brody (Inactive) [ 09/Feb/16 ]

kaloian.manassiev, you're right, it may be possible to make the indexes build in the background on the destination shard, but there are other problems with background index builds on replica sets, such as if there are multiple indexes on any of the collections being transferred replication to the secondaries will wind up blocked as they can't build more than one background index at a time.

So to really make this process work without pain for the user requires a rethinking of how we manage indexes in a sharded cluster, as the workaround for single replica sets (building the index in a rolling fashion across replica set members) doesn't work for any index builds being driven by the system rather than by the user.

Comment by James Wahlin [ 09/Feb/16 ]

kaloian.manassiev - calling MultiIndexBlock::allowBackgroundBuilding() would be an improvement, but I think we will a way to ensure background index builds (either via movePrimary or operationally) to make this safe for production environments that cannot tolerate downtime.

Comment by Kaloian Manassiev [ 09/Feb/16 ]

james.wahlin, the way I understand the problem is that during cloning of collections, because MultiIndexBlock::allowBackgroundBuilding is not called, even if all indexes on the source are background indexes, we will still build them as foreground. Is this correct?

This should be easy to fix by just calling MultiIndexBlock::allowBackgroundBuilding. It won't help if you have mixed foreground and background indexes, but it would make it better in the case where all indexes are background.

spencer, what are your concerns with doing this?

Comment by Spencer Brody (Inactive) [ 08/Feb/16 ]

This won't be easy to fix anytime soon, for now documenting it is probably the best we can do until we have a better overall story for index maintenance in a sharded cluster. Filed DOCS-7112.

Comment by James Wahlin [ 07/Feb/16 ]

The clone command is also impacted by this. movePrimary uses the clone command to transfer data and build indexes.

Generated at Thu Feb 08 04:00:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.