[SERVER-26590] No primary, insert through mongoS hangs for 15 seconds before failing Created: 12/Oct/16  Updated: 20/Dec/16  Resolved: 13/Oct/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alessandro Gherardi Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-21272 All NotMaster and NetworkError retrie... Closed
Operating System: ALL
Steps To Reproduce:
  • Setup a Mongo 3.2.10 cluster with CSRS and 1 replicaset consisting of a single mongoD
  • Create a sharded collection
  • Connect to the mongoD and run db.getSiblingDB("admin").runCommand( {replSetStepDown: 600, force: true}

    )

  • Connect to mongoS
  • Set rs.slaveOk()
  • Try to insert a document in the collection
  • The insert hangs for about 15 seconds then fails with "could not contact primary for replica set"
Participants:

 Description   

In Mongo 3.2.10, when trying to insert a new document via mongoS and the underlying replicaset has no primary, the write hangs for about 15 seconds before failing with a "could not contact primary for replica set" error.

The error is expected. What's not expected is that it takes 15 seconds for mongoS to return that error. In Mongo 3.0.4, the insert fails immediately.



 Comments   
Comment by riccardo salzer [ 20/Dec/16 ]

hi kal,
thanks a lot for your answer, we will update our cluster then in order to avoid this problem in the future.
I like the "giving a chance" in general, but not on every query but only in the intermediate situation when a failover happens and afterwards holding the state somewhere.
thanks, riccardo

Comment by Kaloian Manassiev [ 19/Dec/16 ]

Hi salzamt,

Unfortunately in version 3.2 there is no way to lower this wait time - it was added intentionally as a way of "giving a chance" to the shard's replica set to elect a primary or for the network condition to recover itself.

In version 3.4 we added support on MongoS for enforcing maxTimeMS. If you specify a value lower than 15 seconds, the server will keep looking for primary for at most this value.

Hope this helps.

-Kal.

Comment by riccardo salzer [ 18/Dec/16 ]

Hi,
I just found this ticket because of investigations regarding a downtime on our production servers. Even though this is the expected behaviour, its a very bad practice to hold the response for some time even though you already know, it might fail. This will cause a lot of web applications to go down whenever a network error occurs if they have quiet some request throughput.

Imagine following scenario:

  • dns issue between only two nodes and only one direction.
  • mongos just cannot connect to one single primary of all primaries in the cluster and waits for 15 seconds.
  • in average, lets say with 500 requests/s, 500ms avg request time and 40 workers (uwsgi, pymongo, sharded cluster) working very fine. just 50 or the 500 requests per second would like to write to this one primary. instead of failing, the wait for 15 seconds so in like half a second, all workers are busy with waiting and all requests are queued in the request queue and timing out. nothing works, everything just waits.

actually none really cares if those few requests would raise an exception but everything besides that would at least work. if those are very important, it needs to be handled in the code anyway.
so we really need to disable this "feature" otherwise we will go offline whenever a network issue occurs because of timeouts and blocked requests.
is there a way to disable or set this timeout manually?
also this state of blocking for 15 seconds makes sense in the first 15 seconds after a network issue occurs to wait for a new primary probably, but not forever, if there is no primary (actually there is, mongos is just not seeing it due to network and DNS issues) it should keep a state of the problem and and immediately return an error, not just check and hold on request / query time.
looking forward to a solution

Comment by Kelsey Schubert [ 13/Oct/16 ]

Hi alessandro.gherardi@yahoo.com,

This is expected behavior and was implemented by SERVER-21272. For shards backed by a replica set (including the config server), we will try for up to 15 seconds to locate a primary node and if none is found will fail.

Kind regards,
Thomas

Comment by Kelsey Schubert [ 12/Oct/16 ]

Hi alessandro.gherardi@yahoo.com,

Thanks for reporting this behavior. We are investigating this issue and will update this ticket when we know more.

Kind regards,
Thomas

Generated at Thu Feb 08 04:12:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.