[SERVER-20222] Index rebuilding on secondary during initial resync is not done in parallel for multiple collections or DBs Created: 31/Aug/15  Updated: 26/Sep/15  Resolved: 26/Sep/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication, WiredTiger
Affects Version/s: 3.0.5
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Baruch Oxman Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-9135 fast index build for initial sync Closed
Participants:

 Description   

Some setup info:
Mongo DB primary: 2.6.3
Mongo DB secondary: 3.0.5
Secondary uses wiredTiger

When running an initial resync for an instance that contains 1000+ DBs, each with ~100 collections, with over 1000 indexes overall (per-db), the resync process is very slow. Specifically, the index creation process is very slow.
It seems that the index creation is performed one-by-one, without any parallelization between the collections or between DBs.

It would be wise to rebuild many indexes in parallel, to fully utilize the available resources and shorten the resync process.



 Comments   
Comment by Baruch Oxman [ 01/Sep/15 ]

Thanks Dan,
These other tickets seem to cover most of it.
The situation described in SERVER-19766 could actually be related - I'm gonna check that.

An additional improvement suggestions:

  • Start rebuilding indexes on the secondary as soon as the secondary is done with syncing a DB's / collection's data, and in parallel with syncing of the next DB/collection - this would allow creating a "pipeline" mode, with a shorter overall delay, as in my usecase most DBs do not change when later the oplog is reapplied on the secondary.
  • Have a setting that would control the replication mode. For example, if I'm willing to make a maintenance windown during which the primary is available only for the means of replication, aborting all writes to it, allow some faster replication method, with more performance impact on the primary.
Comment by Daniel Pasette (Inactive) [ 01/Sep/15 ]

Hi Baruch,
The index rebuilding on secondaries during initial sync is done in parallel in v3.0. This issue was resolved with SERVER-9135.

There are a number of ways we're planning to improve the speed of initial sync. Here's a few to watch: SERVER-676, SERVER-18041and SERVER-16206.

Generated at Thu Feb 08 03:53:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.