[SERVER-77008] Parallelize Oplog Fetching Created: 10/May/23 Updated: 29/Aug/23 Resolved: 29/Aug/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kevin Arhelger | Assignee: | Shameek Ray |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Service Arch
|
||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
In cross multi-region replica sets, large geographical distances and low internet bandwidth between regions can lead to excessive replication lag. The aggregate bandwidth between multiple connections is much higher than that of a single socket. Parallelizing oplog fetching to utilize multiple connections could allow linear scaling and reduced replication lag in these scenarios. |
| Comments |
| Comment by Phoebe Du [ 29/Aug/23 ] |
|
We turned this ticket in to PM-3481 |
| Comment by Andy Schwerin [ 23/May/23 ] |
|
I agree with amirsaman.memaripour@mongodb.com. MongoDB replication depends on in-order delivery of oplog entries, and it is not possible to detect gaps in the oplog timestamps on the recipient. That makes a project like this substantially complex at this layer of the system. |
| Comment by Amirsaman Memaripour [ 23/May/23 ] |
|
IIUC, we are proposing using multiple TCP connections to handle the bandwidth limits on cross-region connections. Focusing on the oplog fetcher, this implies we want to collect OpMsg objects over multiple connections on the receiver side, fixing the order when they are received out-of-order, and then delivering them to the higher-level (e.g., replication) as if it was received over a single connection. This requires considerable engineering work, and I'm wondering if we can focus on alternative solutions, like addressing this outside of mongod processes? One possibility is having a transport-level load-balancer where each physical connection from Mongo is translated into multiple cross-region connections. Since we are targeting a specific deployment, that might be a better solution, in terms of development, maintenance, and complexity costs? |