[SERVER-8990] Support Replication Between Sharded Clusters Created: 15/Mar/13  Updated: 06/Dec/22  Resolved: 07/Oct/19

Status: Closed
Project: Core Server
Component/s: Replication, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Vasily Assignee: Backlog - Replication Team
Resolution: Won't Do Votes: 5
Labels: pull-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File img1.png     PNG File img2.png     PNG File img3.png    
Issue Links:
Related
related to SERVER-2956 Master-Master replication Closed
Assigned Teams:
Replication
Participants:

 Description   

Please support replication between sharded clusters. This would effectively first require multi-master replication behavior and support to exist.

One Idea:
Good idea to use an existing op-log for cross-cluster replication. Read from op-log from each shard and apply the operation on destination mongos (another datacenter).

The problem is in the same operation will appear in destination cluster op-log and infinite loop will start as a result.

The solution is to append to document a special marker to specify the appurtenance to the particular cluster.



 Comments   
Comment by Steven Vannelli [ 07/Oct/19 ]

Closing this ticket as Won't Do as the parent Epic is no longer needed at this time.

Comment by Ramon Fernandez Marina [ 17/Jan/15 ]

vasmik, thanks for taking the time to produce a pull request, but unfortunately we cannot merge it at this time: adding a "payload" to every oplog entry may have a huge performance impact and other ill effects on existing deployments, so this is a change that would need very careful consideration and thorough field testing. We're therefore closing your pull request for now, but keeping the ticket open for future consideration.

Note that you may insert markers in your oplog at any time, as per SERVER-12119, provided you use MongoDB version 2.6 or newer.

Regards,
Ramón.

Comment by Vasily [ 18/Mar/13 ]

Hi Scott,
We understand that we can have a shard key which is DC specific. But if we have data written in one DC we will not be able to modify them in another DC locally. We understand that we can potentially update data from within DC2 directly to DC1. But if DC1 is down these data will not be modifiable. This is the problem which we, and we believe MongoDB in general, should address. Otherwise MongoDB is inferior to rival Cassandra, CouchBase or other NoSQL DBs. Adding DC marker will make op-log DC-aware, allowing other developers to make their own replication.

About conflict resolution. Other DB providers, e.g. Cassandra, CouchBase etc, solved this problem one way or another. There is industry best practices to resolve conflicts and they can be implemented in replicator app. It could be totally independent from MongoDB. But MongoDB would have a way to make this replication possible if op-log would contain DC marker.

Comment by Vasily [ 15/Mar/13 ]

Actually we have an external sync-engine that uses this marker. This tool is responsible to conflict-resolution. SERVER-2956 issue exists for almost 2 years without any solution as I can see.
My pull-request: https://github.com/mongodb/mongo/pull/400

Comment by Scott Hernandez (Inactive) [ 15/Mar/13 ]

What you are essentially asking for is multi-master replication (see linked issue) plus replication between two sharded clusters. As you have shown, you can build something like this external to our current sharded clusters and replication system (using the same type of architecture), but it is not something which is built-in today.

Do you ever have the chance in your application to write data which will conflict? What would expect to happen if there was a conflict and the data had to be merged or replaced based on some policy? No need to answer here, but these questions are crux of the multi-master replication challenge and would probably be best discussed in that feature request.

Currently you can use a single sharded cluster with different shards which have a primary in each data center to make sure that writes (esp. easy with inserts) are always local to the data center; you basically write to the master in your data center. To do this you need to combine a synthetic shard key, if you don't already have one which isolates writes to the datacenter/region, and shard/chunk tags to keep the data distributed (balanced on) to the data center you wish to write to.

Comment by Vasily [ 15/Mar/13 ]

"Vasily, this sounds like you want to do replication between two separate sharded clusters across different groups of replica sets in each sharded cluster. Is that the feature you are asking for?"

Exactly! Because of MongoDB does not support master-master replication by default we can't share the replica-sets between datacenters (each RS has nodes in both DCs).
But we wish to get an ability to make writes to both DCs simultaneously. It looks like a bi-directional replication or master-master replication.
In this case we can set independent clusters for each DC in different configuration.

Thanks for your attention, Scott!

Comment by Scott Hernandez (Inactive) [ 15/Mar/13 ]

Vasily, this sounds like you want to do replication between two separate sharded clusters across different groups of replica sets in each sharded cluster. Is that the feature you are asking for?

Currently nothing like this is supported by mongodb and the "replication servers" you have listed in your diagrams aren't a component we have. Can you explain a little more the use-case you are trying to address?

Generated at Thu Feb 08 03:19:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.