[SERVER-22503] movePrimary builds indexes in foreground on destination shard Created: 07/Feb/16 Updated: 06/Dec/22 Resolved: 09/Dec/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | 3.0.9, 3.2.1 |
| Fix Version/s: | 4.2.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | James Wahlin | Assignee: | [DO NOT USE] Backlog - Sharding Team |
| Resolution: | Done | Votes: | 2 |
| Labels: | pm-1051-legacy-tickets | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Sharding
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Description |
|
The movePrimary command is used to move primary database and unsharded collections from one shard to another. When movePrimary clones a collection from the donor to the recipient shard, it will create indexes using a MultiIndexBlock. It does not call MultiIndexBlock::allowBackgroundBuilding() before executing. Without this call all index builds will be performed in the foreground, regardless of whether {background: true} is set in a given index's specification. Performing foreground index builds on the recipient shard will block all database read/write activity for the duration of each index build. This can be a significant period of time depending on data size and hardware resources available. We should look to build indexes in the background during movePrimary. If not possible, we should at a minimum update the movePrimary documentation to reflect. This also impacts CloudManager shard removal via automation, in the case where the shard being removed is the primary database for unsharded collections. For movePrimary index build code see: |
| Comments |
| Comment by Sheeri Cabral (Inactive) [ 09/Dec/19 ] |
|
fixed in 4.1.8. Hybrid Index builds project fixed the negative impact. |
| Comment by Eric Milkie [ 14/Jun/18 ] |
|
It turns out that movePrimary clones all the data and then builds indexes, similar to initial sync. So one way we could fix this is by building the indexes first and then cloning the data. |
| Comment by Eric Milkie [ 22/May/18 ] |
|
Putting this ticket back in Needs Triage, as I don't believe this is a problem, at least in 4.0. I would expect the movePrimary command to build indexes on empty collections, not ones with data in them. |
| Comment by Spencer Brody (Inactive) [ 09/Feb/16 ] |
|
kaloian.manassiev, you're right, it may be possible to make the indexes build in the background on the destination shard, but there are other problems with background index builds on replica sets, such as if there are multiple indexes on any of the collections being transferred replication to the secondaries will wind up blocked as they can't build more than one background index at a time. So to really make this process work without pain for the user requires a rethinking of how we manage indexes in a sharded cluster, as the workaround for single replica sets (building the index in a rolling fashion across replica set members) doesn't work for any index builds being driven by the system rather than by the user. |
| Comment by James Wahlin [ 09/Feb/16 ] |
|
kaloian.manassiev - calling MultiIndexBlock::allowBackgroundBuilding() would be an improvement, but I think we will a way to ensure background index builds (either via movePrimary or operationally) to make this safe for production environments that cannot tolerate downtime. |
| Comment by Kaloian Manassiev [ 09/Feb/16 ] |
|
james.wahlin, the way I understand the problem is that during cloning of collections, because MultiIndexBlock::allowBackgroundBuilding is not called, even if all indexes on the source are background indexes, we will still build them as foreground. Is this correct? This should be easy to fix by just calling MultiIndexBlock::allowBackgroundBuilding. It won't help if you have mixed foreground and background indexes, but it would make it better in the case where all indexes are background. spencer, what are your concerns with doing this? |
| Comment by Spencer Brody (Inactive) [ 08/Feb/16 ] |
|
This won't be easy to fix anytime soon, for now documenting it is probably the best we can do until we have a better overall story for index maintenance in a sharded cluster. Filed |
| Comment by James Wahlin [ 07/Feb/16 ] |
|
The clone command is also impacted by this. movePrimary uses the clone command to transfer data and build indexes. |