[SERVER-3921] Add command to force a secondary to sync from a specific replica set member Created: 21/Sep/11 Updated: 19/Nov/16 Resolved: 10/Oct/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 2.1.1 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Kyle Banker | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
This is necessary in high-load situation, where you might want to resync from a hidden member, for example. |
| Comments |
| Comment by Shakir Sadikali [ 07/Oct/16 ] |
|
OK, I just found |
| Comment by Shakir Sadikali [ 07/Oct/16 ] |
|
I'm requesting that we reopen this request. I've had numerous requests in Support with customers asking how they can force an initial sync from a designated secondary (or any node for that matter). All from MongoDB 3.0 onwards. While I appreciate that we can specify the sync source... the mechanics of doing so are painful (with multiple customers using the term "hacky" to describe it). Can we have a more elegant solution to it? Perhaps being able to add in the sync source when actually adding the node or a startup parameter to control it when it comes up for the first time? I appreciate this may not be the right place to put this request. Please direct me to the right place. Cheers, |
| Comment by auto [ 22/Mar/12 ] |
|
Author: {u'login': u'kchodorow', u'name': u'Kristina', u'email': u'kristina@10gen.com'}Message: Added replSetSyncFrom command Conflicts: src/mongo/db/repl/rs_sync.cpp |
| Comment by Jason Koppe [ 20/Jan/12 ] |
|
Using tags seems good to me. Our replica sets span multiple datacenters and it'd be extremely useful to sync from a specific tag (rack, datacenter, etc).
I don't think you should fallback to syncing from a member we didn't specify. A list of tags to try seems good, and if it cannot sync from any of those servers, just stay in recovering state with some error message. |
| Comment by eli jones [ 21/Nov/11 ] |
|
Kristina, Tags could work nicely. If it is simpler on your end to have tagging specify replica sync behaviour (instead of using an optional command line parameter), then I'm very happy to use tagging. So.. for our purposes, we could just have a "non-hidden" tag (or "East Coast Sync-able"). And, if the replica set is in a healthy state, at least one server tagged as "non-hidden" will be available at any given time. Then, you could just use your regular initial sync heuristic against those servers to pick which one to sync to. Having an array of tags would be nice.. since could have an "East Coast Sync-able" tag and then a "West Coast Sync-able" one. Though, in the end.. I personally prefer a command line parameter.. since at the point in time that I am adding a new member or resyncing an existing member.. it's hard to say for sure what state the architecture will be in. And, at that moment.. I will know that I need the member to sync to the Primary.. since, for example maybe there is a secondary that it could sync to.. but that secondary's data is not in RAM.. Maybe the secondary's data has become sort of fragmented and big.. and if the initial oplog application pulls from it.. that will push too much data out of RAM (to make room for the oplog reads).. and cause me heartache.. etc. But, if this tagging method is the compromise, I am very happy to get it. Thanks again for all your help with this. |
| Comment by Kristina Chodorow (Inactive) [ 21/Nov/11 ] |
|
@Ian: Tagging lets you specify any set of servers, so you could use it for syncing from a specific server, syncing from a certain rack, etc. See http://www.mongodb.org/display/DOCS/Data+Center+Awareness#DataCenterAwareness-Tagging%28version2.0%29. I like the idea of using tags, probably in the replica set config (e.g., {_id : 0, host : ..., syncFrom : "tag1"} or something). What if the server(s) we ask a member to sync from aren't available, though? Should it just sit there and not sync? Sync from a member we didn't specify? That could suddenly put load on the primary or something. Maybe allow an array of tags, first try servers tagged "x", then "y", then "z" ({_id : 0, host : ..., syncFrom : ["x", "y", "z"]}. |
| Comment by Ian White [ 20/Nov/11 ] |
|
We would be excited to get this ability in any form, but it seems much more awkward to me if it's a command. When you start a node that is already a replset member with empty data, it starts up and then immediately starts syncing automatically. It seems like it would be awkward to start up mongod, and then have to quickly enter into the shell and execute a command before it starts syncing. But maybe I'm misunderstanding. Re: tagging – how this would work? Is the idea that we would tag datacenters and then be able to specify a tag to sync from, but not the specific node? |
| Comment by Dwight Merriman [ 17/Nov/11 ] |
|
if we ever did have an option, i'd like it to not be a cmd line parameter but something else. not sure what makes sense. could be via the setParameter command if it is something rarely used. ideally have no setting. hmmm. given we have tagging could it be done via that? |
| Comment by eli jones [ 24/Sep/11 ] |
|
I would like to reiterate Ian's point (granted, we work together so I guess this is just double voting).. whatever heuristic you use to pick a node to sync to.. it will not be able to cover all scenarios. So, in the event of a situation where the heuristic fails and an administrator wants to sync to a specific node that isn't hidden (or isn't a secondary), it would be nice to have an optional --syncTo flag. |
| Comment by Ian White [ 23/Sep/11 ] |
|
Not a fan of this idea. This heuristic seems to assume that a hidden node is going to always be lower load than non-hidden nodes. But in many architectures, people have backup nodes that are running on underpowered pieces of hardware. They're hidden because they're backups, and nobody is supposed to query them if it's avoidable, not even their oplog. In those scenarios, you're better off syncing from something else. (This is definitely true of our current architecture – we have most of our sets writing to a hidden backup node which is usually very busy. We use it just to take snapshots and do emergency recovery. We generally want to avoid syncing from it.) If you want a general heuristic, it would be better I think to measure load on the nodes and prefer syncing from the node with lowest load. Whatever heuristic you settle on, though, having the option to manually override with a --syncto parameter would be appreciated for the edge cases where the heuristic fails. Otherwise we're put in the position of having to shut down nodes to get it to sync the right one. |