[SERVER-20517] Queries failing when replica set secondary is down. Created: 21/Sep/15  Updated: 06/Apr/23  Resolved: 23/Sep/15

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.0.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Scott Lowe Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian7


Operating System: Linux
Steps To Reproduce:

Stop secondary
Observer timing out queries, and errors in mongos logs.

Participants:

 Description   

We have a sharded cluster, with mongos on each of the app servers. If I stop a secondary on the first shard, then from the replica set point of view, everything seems ok (primary still up). However, our application servers are then unable to query the local mongos (the queries just time out).

Looking at the mongos logs, there are errors like this:

W NETWORK [Balancer] Failed to connect to <IP of replica set SECONDARY>, reason: errno:111 Connection refused
W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to <IP of replica set SECONDARY>, reason: errno:111 Connection refused



 Comments   
Comment by Scott Lowe [ 23/Sep/15 ]

Doh, someone had slipped a read preference in the app code. It's working as expected now.

Thank you for your support, and sorry for the false issue.

Comment by Sam Kleinman (Inactive) [ 21/Sep/15 ]

Thanks for this report, and sorry that you're experiencing this issue. I have a few questions about your configuration and system:

  1. What read preference or preferences are your applications using?
  2. Can you provide a copy of the output of sh.status() while connected to the mongos systems?
  3. Do you see the same behavior while connected to all mongos instances?
  4. Do all app servers have identical workloads, or are some specialized for worker or reporting roles?
  5. How do you resolve this issue? Does the issue go away after a period of time? Are you restarting your mongos instances?

Regards,
sam

Generated at Thu Feb 08 03:54:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.