[SERVER-7527] Improve speed of replication when in initial sync Created: 01/Nov/12 Updated: 08/Jan/23 Resolved: 14/Jun/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andre de Frere | Assignee: | Backlog - Replication Team |
| Resolution: | Done | Votes: | 7 |
| Labels: | PM248 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Replication
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Full sync using mongo is VERY slow. The transfer rates obtained via multi-session rsync with compression are many times faster and could be indicative in where improvements could be made. As it stands, nodes will be in initial sync for excessive amounts of time on anything other than a moderate amount of data. |
| Comments |
| Comment by Alan Zheng [ 06/Jan/23 ] |
|
Unfortunately, this is only for Enterprise Edition. We currently do not have plans to make this available for community. |
| Comment by Oded Raiches [ 05/Jan/23 ] |
|
Thanks for the fast response Alan! I see that this option is Available in MongoDB Enterprise only. Is there an option for community edition users? |
| Comment by Alan Zheng [ 04/Jan/23 ] |
|
Since MongoDB 5.2 Rapid Release, we've introduced File Copy Based Initial Sync. With this method, MongoDB will copy files from the file system of the source node to the file system of the target node. This process can be faster than a logical initial sync, especially at larger data sizes. In our testing, initial sync via file copy was nearly four times (4X) faster than a logical initial sync on the same dataset. You can read more from our Blog Post. |
| Comment by Oded Raiches [ 04/Jan/23 ] |
|
Hi, along side the slowness in initial-sync, having some other weird issues. I have a primary which has ~340GB of data that needs to be synced. The syncing node is very slow but sometimes even drops in its capacity and not being able to get back in order, just stuck in STARTUP2 for a huge amount of time. How are you progressing with improving the initial-sync speed, and what are your production recommendations for making it faster currently? using mongo 6.0.3 Adding zip file of the logs in the syncing node: [^mongologs.zip] |
| Comment by Spencer Brody (Inactive) [ 14/Jun/18 ] |
|
We have plans to run a whole project around improving initial sync performance |