[SERVER-7527] Improve speed of replication when in initial sync Created: 01/Nov/12  Updated: 08/Jan/23  Resolved: 14/Jun/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Andre de Frere Assignee: Backlog - Replication Team
Resolution: Done Votes: 7
Labels: PM248
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-3160 replication initial sync should use t... Closed
Related
related to SERVER-18041 Support parallel cloning during initi... Investigating
is related to SERVER-14805 Use multithreaded oplog replay during... Closed
Assigned Teams:
Replication
Participants:

 Description   

Full sync using mongo is VERY slow. The transfer rates obtained via multi-session rsync with compression are many times faster and could be indicative in where improvements could be made.

As it stands, nodes will be in initial sync for excessive amounts of time on anything other than a moderate amount of data.



 Comments   
Comment by Alan Zheng [ 06/Jan/23 ]

Unfortunately, this is only for Enterprise Edition. We currently do not have plans to make this available for community. 

Comment by Oded Raiches [ 05/Jan/23 ]

Thanks for the fast response Alan!

I see that this option is Available in MongoDB Enterprise only.

Is there an option for community edition users?

Comment by Alan Zheng [ 04/Jan/23 ]

Since MongoDB 5.2 Rapid Release, we've introduced File Copy Based Initial Sync

With this method, MongoDB will copy files from the file system of the source node to the file system of the target node. This process can be faster than a logical initial sync, especially at larger data sizes. In our testing, initial sync via file copy was nearly four times (4X) faster than a logical initial sync on the same dataset. You can read more from our Blog Post.

Comment by Oded Raiches [ 04/Jan/23 ]

Hi, along side the slowness in initial-sync, having some other weird issues.

I have a primary which has ~340GB of data that needs to be synced.

The syncing node is very slow but sometimes even drops in its capacity and not being able to get back in order, just stuck in STARTUP2  for a huge amount of time.

How are you progressing with improving the initial-sync speed, and what are your production recommendations for making it faster currently? using mongo 6.0.3

Adding zip file of the logs in the syncing node:

[^mongologs.zip]

Comment by Spencer Brody (Inactive) [ 14/Jun/18 ]

We have plans to run a whole project around improving initial sync performance

Generated at Thu Feb 08 03:14:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.