[SERVER-13244] Secondary that enters RECOVERY mode does not reset connections Created: 18/Mar/14  Updated: 06/Dec/22

Status: Open
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: Needs Further Definition

Type: Bug Priority: Major - P3
Reporter: Andrew Ryder (Inactive) Assignee: Backlog - Replication Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Assigned Teams:
Replication
Operating System: ALL
Participants:

 Description   

When a node changes state normally (most notably from an election) it disconnects all sockets partly to ensure the change of state is noticed. However, when a Secondary enters RECOVERY mode due to a compact command, it does not close existing sockets.

Clients using secondaryPreferred continue to utilize the node as a normal secondary (operations either get blocked or return a failure due to "not primary or secondary") but new clients do not attempt to issue queries because the invalid state is detected.

If the change in state means that normal client operations cannot proceed, the sockets should be reset, as is the case for elections.



 Comments   
Comment by Andrew Ryder (Inactive) [ 09/Apr/14 ]
  1. When a server transitions from PRIMARY to SECONDARY, future write commands will fail.
  2. When a server transitions from SECONDARY to RECOVERING, future read commands will fail.

The reason for primary to non-primary connection dropping is because of writes (just think about the wire protocol), which is a special case and fairly important.

The wire protocol has no concept of node states.

The insert command is grammatically identical to the getLastError command, and is encoded into BSON the same way. One of these is write-only. The other is read-only. The wire protocol cannot tell the difference. The availability of these commands is determined by the state of the instance one is connected to - nothing to do with the wire protocol.

I do not see what is special about writing that does not apply to reading. The impairment of either function is of equal importance. A database that cannot be written is useless. A database that cannot be read is useless.

But why not simply an error message for both conditions then? Why an error message for one, and abrupt disconnection for another?

Comment by Scott Hernandez (Inactive) [ 24/Mar/14 ]

Drivers/clients do periodically pull/check the state of each node, and should react when the server responds with not-master/slave-ok. The reason for primary to non-primary connection dropping is because of writes (just think about the wire protocol), which is a special case and fairly important.

It is not a general rule to drop connection on state changes, no – making it so would dramatically change client/system behavior.

Comment by Andrew Ryder (Inactive) [ 24/Mar/14 ]

I agree the behavior could be addressed at the driver, but that doesn't really address my question.

I want to clarify:
Do drivers need to actively monitor for state changes or not?
If yes, then why disconnect existing clients on PRIMARY / SECONDARY transitions?
If no, then why not disconnect existing clients on other state changes that affect the outcome?

I am inquiring as to what is special about the PRIMARY / SECONDARY transition such that it warrants dropping existing connections but which does not apply to other state changes.

If the behavior to "drop all connections" when changing between SECONDARY and PRIMARY conditions is for a different reason (that is, if the change in state is incidental to the rationale behind terminating all connections) then the retention of connections through the transition to RECOVERING state can be justified as having no special significance. Right now, the SECONDARY / PRIMARY transition highlights a connection retention rule that does not appear to be consistently applied.

Is there a rule being applied that I do not understand?

Comment by Eric Milkie [ 18/Mar/14 ]

Is there a specific driver you had in mind for this behavior? I'm not sure that a blanket statement that all clients using secondaryPreferred would indefinitely produce errors when a particular node in a set is in state RECOVERING is accurate. In general, I believe this is an issue that can improved at the driver level, if it isn't already optimal. After receiving one error, it's certainly possible for the client to rescan the state of the replica set and avoid sending further read requests to a RECOVERING node.

Generated at Thu Feb 08 03:31:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.