[SERVER-77008] Parallelize Oplog Fetching Created: 10/May/23  Updated: 29/Aug/23  Resolved: 29/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Arhelger Assignee: Shameek Ray
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Related
Assigned Teams:
Service Arch
Participants:
Case:

 Description   

In cross multi-region replica sets, large geographical distances and low internet bandwidth between regions can lead to excessive replication lag. The aggregate bandwidth between multiple connections is much higher than that of a single socket.

Parallelizing oplog fetching to utilize multiple connections could allow linear scaling and reduced replication lag in these scenarios.



 Comments   
Comment by Phoebe Du [ 29/Aug/23 ]

We turned this ticket in to PM-3481

Comment by Andy Schwerin [ 23/May/23 ]

I agree with amirsaman.memaripour@mongodb.com. MongoDB replication depends on in-order delivery of oplog entries, and it is not possible to detect gaps in the oplog timestamps on the recipient. That makes a project like this substantially complex at this layer of the system.

Comment by Amirsaman Memaripour [ 23/May/23 ]

IIUC, we are proposing using multiple TCP connections to handle the bandwidth limits on cross-region connections. Focusing on the oplog fetcher, this implies we want to collect OpMsg objects over multiple connections on the receiver side, fixing the order when they are received out-of-order, and then delivering them to the higher-level (e.g., replication) as if it was received over a single connection. This requires considerable engineering work, and I'm wondering if we can focus on alternative solutions, like addressing this outside of mongod processes? One possibility is having a transport-level load-balancer where each physical connection from Mongo is translated into multiple cross-region connections. Since we are targeting a specific deployment, that might be a better solution, in terms of development, maintenance, and complexity costs?

Generated at Thu Feb 08 06:34:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.