[SERVER-24688] Graceful shutdown without destroying the current cmd Created: 21/Jun/16  Updated: 08/Feb/23  Resolved: 15/Jul/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Tommaso Allevi Assignee: Tess Avitabile (Inactive)
Resolution: Duplicate Votes: 0
Labels: move-sa, platforms-re-triaged
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-46958 Create server parameter shutdownTimeo... Closed
Sprint: Repl 2020-07-27
Participants:

 Description   

This feature would provide the option to request a graceful shutdown of mongos. Its behavior is outlined below:

  1. the client sends a request to a mongos that forwards the query to mongodb
  2. when mongos receives a specific signal (like SIGTERM), all heartbeats return an error to the client
  3. the client removes that mongos from the available mongos pool and does not send any new requests to that mongos
  4. mongos waits until all old queries are done without accepting new ones
  5. mongos now is free to shut down itself
    In the fourth item, mongos could wait endlessly, so a timer can force the shutdown aggressively to avoid stall state.
Original Description

Is it possible to shutdown mongos daemon gracefully? Which signal I should launch to the process?

Thinking about a mongos that has received a query. While the mongodb calculates the query, the mongos can be stopped. If mongos is stopped, the client driver (like nodejs does) returns a socket error.
This mean that the client is not able to know if the query runs successfully or not.
In a write context is a problem. A socket error is undetermined state.



 Comments   
Comment by Tess Avitabile (Inactive) [ 15/Jul/20 ]

Hi allevo,

Thank you for this feature request. In response to requests for a more graceful shutdown on mongod and mongos, we've implemented the quiesce mode feature:

mongod secondaries and mongos will enter a quiesce mode prior to shutdown, to allow short-running operations to finish. During this time, new and existing operations are allowed to run, but isMaster requests return a ShutdownInProgress error, to indicate that clients should start routing operations to other nodes. On mongod, quiesce mode happens after the stepdown attempt, and only if the node is now in state SECONDARY. The combined time for stepdown and quiesce mode is specified by the timeoutSecs parameter to the shutdown command (for command-based shutdown) or the shutdownTimeoutMillisForSignaledShutdown/mongosShutdownTimeoutMillisForSignaledShutdown server parameter (for signaled shutdown).

This work is completed as of SERVER-46952 and SERVER-46958. I believe this satisfies your original ticket, so I'm going to closed this as a duplicate of SERVER-46958. If not, please reopen this request and provide more details about your requirements.

Kind regards,

Tess

Comment by Kelsey Schubert [ 22/Jun/16 ]

Hi allevo,

Thank you for clarifying this feature request. I've updated the ticket description to reflect your latest comment and have marked this ticket to be considered during the next round of planning. Please continue to watch for updates

Kind regards,
Thomas

Comment by Tommaso Allevi [ 22/Jun/16 ]

The problem is clear: according to the theory, in a production environment, when the high availability is required, the application has to trust in its database. No undefined behavior should be permitted.
Practically, this can never happen. So the chosen behavior should reduce as lower as possible the nondeterministic behavior.

I'm thinking when our DBA team is upgrading mongos cluster.

I suppose the expected behavior during a graceful shutdown is like this:

  1. the client sends a request to a mongos that forward the query to mongodb
  2. when mongos receives a specific signal (like SIGTERM), all heartbeats return an error to the client
  3. the client remove that mongos from the available mongos pool and shouldn't send new request to that mongos
  4. mongos wait until all old queries are done without accepting new ones
  5. mongos now is free to shut down itself
    In the fourth item, mongos cloud wait endlessly, so a timer can force the shutdown aggressively to avoid stall state.
Comment by Ramon Fernandez Marina [ 21/Jun/16 ]

allevo, what's the behavior you think would make sense? If the mongos is shut down the client can no longer receive any information from the server.

We can turn this ticket into a feature request, but it needs a better definition of the expected behavior, so can you please provide additional details?

Thanks,
Ramón.

Generated at Thu Feb 08 04:07:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.