[SERVER-25205] MongoDB overwrites version field in replSetReconfig if "force" is given Created: 21/Jul/16  Updated: 13/Aug/16  Resolved: 02/Aug/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Joseph Glanville Assignee: Unassigned
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Run replSetReconfig with an explicit version in the config document
Note in the replSetStatus it is actually a much larger and random number.

Participants:

 Description   

When trying to explicitly set the version of the replica set configuration the server will overwrite the version specified by the operator if the "force" flag is given.

Instead the version number will be incremented randomly by a relatively large number.

This behaviour is undesirable if one intends to build a system to automatically update the replica set configuration as in these cases it's important the external system is able to set the version to a known number. It is the job of that external system to ensure it's monotonically increasing etc.

Ideally the behaviour would be changed to not overwrite operator input and instead like the non-force path should return an error if the version number is less than the old version.

If I have missed something vital that explains why the server does this I would love to hear the rational behind this behaviour.



 Comments   
Comment by Joseph Glanville [ 13/Aug/16 ]

I see this has been closed as Works As Designed but I think the problem here is that this behavior eliminates the possibility of a fully safe automated orchestration system for MongoDB without compiling a custom build without this random increment.

I posted this to the list but didn't get any additional replies but it summarises my point fairly concisely:

I agree the random increment is probably a useful measure in the case that an administrator is manually triggering a force reconfigure and there is the chance for operator error to issue this against more than one peer.
I also agree that force reconfigure from scripts isn't a good idea for similar reasons, mostly that most scripts do not have complete information.

However my use-case is very different. I have an external system that has complete knowledge of all MongoDB peers, oplog positions, etc and could ensure that only monotonically increasing replSetConfig version numbers are used.

Being able to apply all reconfigures with known version numbers greatly increases the safety of my usecase because otherwise if my system does actually issue a reconfigure using stale information due to a race condition that it will be properly rejected by the cluster.

Without this functionality such a system can't be made fully safe.

Would it be possible to implement a means to circumvent this random increment in the case that you really do understand the consequences?

Comment by Kelsey Schubert [ 02/Aug/16 ]

Hi josephglanville,

I see you also posted this question on the mongodb-dev user group where Dwight provided the rational for this behavior:

the original intent with force:true was for use in exceptional situations, when it isn't possible to establish a primary in a replica set. the idea was that this would be rare and a manual human intervention event, as normally it could be dangerous to reconfigure without a primary as different members could accept different reconfigs from different clients.

the random increment of the version number is to help mitigate this somewhat. if two set members received forced reconfigs, if they both simply incremented their version #, they would end up at the same new version number, but potentially with different configurations.

to say another way, the idea for force:true was that it is for cases where establishing a primary is impossible forever (say if 2 servers were permanently destroyed from a 3 member set). in theory, one might automate that, but it would require a high level of care, and should be rare, and in a situation like the example i'd want to look at the situation manually personally.

To summarize, this is expected behavior and provides some protection against two replica set members receiving forced configs at the same time. We recommend that {force: true} be used only in emergencies and advise against using it as part of a script.

For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by Joseph Glanville [ 21/Jul/16 ]

I couldn't find an edit button but to elaborate on why I need to be able to use the "force" flag is that occasionally the system needs to recover the replica set from states where a quorum of members is no longer available and will often need to be used on non-primary peers.

Generated at Thu Feb 08 04:08:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.