[SERVER-17124] Ability to start a node in maintenance mode Created: 29/Jan/15 Updated: 05/Apr/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Andre de Frere | Assignee: | Alan Zheng |
| Resolution: | Unresolved | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
Under certain conditions, it is preferred to have a node start and join the replica set, but not be the target for any client reads. A config option to set maintenance mode on start would ensure that the member doesn't take any queries until an operator affirmatively verifies it's good to go. For larger deployments with lots of shards & replicas, a few nodes in maintenance mode waiting for manual verification is operationally safer than serving stale data. |
| Comments |
| Comment by David Bartley [ 02/Apr/20 ] |
|
fwiw, we recently switched away from this approach and now reconfigure the replset to make the node under maintenance hidden with priority 0. |
| Comment by David Bartley [ 20/Jul/18 ] |
|
I think that's the ideal solution, but it can be difficult to audit every application that connects to mongo and uses secondary read preference.
For some background, the way we perform long-running maintenance tasks (e.g. index builds) is: 1) Restart node without --replst and on some alternate port (e.g. 9000) 2) Performance maintenance task 3) Update local.replset.minvalid to the current time 4) Restart node with --replset on the standard port (i.e. 27017) 5) adminCommand({replSetMaintenance: 1})
Step 3 ensures that the node will remain in RECOVERING state until it's caught up to minvalid, which gives us ample time to switch the node to maintenance mode.
A less hacky solution might be to allow maintenance mode to be persistent across mongod restarts. |
| Comment by Asya Kamsky [ 07/Dec/17 ] |
|
Note that if clients don't want to read stale data they should use the driver option maxStalenessSeconds to avoid reading from still catching up secondaries. |
| Comment by Asya Kamsky [ 07/Dec/17 ] |
|
One can start up and do replSetMaintenance command but that leaves a window open for clients to connect. Would ability to start up with flag that run replSetMaintenance before coming on line for clients. |
| Comment by Asya Kamsky [ 07/Dec/17 ] |
|
So be able to start up in MAINTENANCE mode by option. |