[SERVER-23192] mongos and shards will become unusable if contact is lost with all CSRS config server nodes for more than 30 consecutive failed attempts to contact Created: 16/Mar/16  Updated: 20/Sep/18  Resolved: 01/Aug/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.2.4
Fix Version/s: 3.3.11

Type: Bug Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: Misha Tyulenev
Resolution: Done Votes: 5
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongodb.log.2016-12-14T15-43-05.gz    
Issue Links:
Depends
Documented
is documented by DOCS-8928 mongos and shards will become unusabl... Closed
Duplicate
is duplicated by SERVER-22971 Operations on some sharded collection... Closed
Related
related to SERVER-22107 Improve error message when ReplicaSet... Closed
is related to SERVER-23345 RAII semantics for ReplicaSetMonitor Closed
is related to SERVER-25516 Add setParameter option to 3.2 to pre... Closed
is related to SERVER-26719 Improve logging when config server do... Closed
Backwards Compatibility: Minor Change
Operating System: ALL
Steps To Reproduce:

See the comment in jstests/sharding/startup_with_all_configs_down.js.

Sprint: Sharding 16 (06/24/16), Sharding 18 (08/05/16)
Participants:
Case:
Linked BF Score: 0

 Description   
Issue Status as of Oct 07, 2016

ISSUE DESCRIPTION AND IMPACT
If mongos loses network contact with all nodes from the CSRS config server set (both primary and secondaries), the replica set monitor will deem this set as as 'unusable' and will stop monitoring it. As a result, all operations which need to access config server metadata will fail.

DIAGNOSIS AND AFFECTED VERSIONS
This issue is present on MongoDB 3.2.0 to 3.2.9.

Operations that require access config server metadata will begin failing with the following error:

> db.foo.find().itcount();
 
2016-03-16T16:59:20.941-0400 E QUERY    [thread1] Error: error: {
        "code" : 71,
        "ok" : 0,
        "errmsg" : "None of the hosts for replica set test-configRS could be contacted."
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DBCommandCursor@src/mongo/shell/query.js:694:1
DBQuery.prototype._exec@src/mongo/shell/query.js:118:28
DBQuery.prototype.hasNext@src/mongo/shell/query.js:281:5
DBQuery.prototype.itcount@src/mongo/shell/query.js:407:12
@(shell):1:16

REMEDIATION AND WORKAROUNDS
To resolve this issue, restart the affected mongos or mongod.

This issue has been fixed in MongoDB 3.4.0 and 3.2.10:

  • MongoDB 3.4.0 contains the fix described in this ticket..
  • MongoDB 3.2.10 contains the fix described by SERVER-25516.

On versions prior to MongoDB 3.2.10, this issue can be avoided by executing the following command at runtime on all mongos and mongod nodes:

db.adminCommand( {setParameter: 1, 'replMonitorMaxFailedChecks': 2147483647} )

Please note that this parameter does not persist and must be set each time the node restarts.

Original description

If mongos loses network contact with all nodes from the CSRS config server set (both primary and secondaries), the replica set monitor will deem this set as as 'unusable' and will stop monitoring it.

From this point onward all operations which need to access some config server metadata will begin failing with the following error:

> db.foo.find().itcount();
 
2016-03-16T16:59:20.941-0400 E QUERY    [thread1] Error: error: {
        "code" : 71,
        "ok" : 0,
        "errmsg" : "None of the hosts for replica set test-configRS could be contacted."
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DBCommandCursor@src/mongo/shell/query.js:694:1
DBQuery.prototype._exec@src/mongo/shell/query.js:118:28
DBQuery.prototype.hasNext@src/mongo/shell/query.js:281:5
DBQuery.prototype.itcount@src/mongo/shell/query.js:407:12
@(shell):1:16

This includes the refresh of the list of shards, which needs to be read from the config server metadata. Therefore, currently there is no procedure to restart or retry monitoring of the config server set and the only recourse is to restart mongos.



 Comments   
Comment by Kay Kim (Inactive) [ 19/Jul/17 ]

Thanks much. Will finally doc this ticket all 9000 years after the fact

Comment by Misha Tyulenev [ 19/Jul/17 ]

kay.kim IMO config databases can be misinterpreted - may be better to say "CSRS condfig server nodes"
Instead of "30 tries" use replMonitorMaxFailedChecks consecutive tries where the default is 30. Then you can specify the remediation as in the ticket

Comment by Misha Tyulenev [ 09/Dec/16 ]

akira.kurogane, Thanks for the update.

Comment by Spencer Brody (Inactive) [ 26/Aug/16 ]

We decided that backporting this to 3.2 is not feasible due to how much the code has changed since then, and decided to do SERVER-25516 as a workaround for 3.2 instead.

Comment by Spencer Brody (Inactive) [ 02/Aug/16 ]

Adding to "Needs Triage" specifically to discuss if we need to do a separate, smaller-scope fix for 3.2

Comment by Misha Tyulenev [ 01/Aug/16 ]

The change has removed the replMonitorMaxFailedChecks parameter as its not applicable anymore. The fix makes replica state monitoring unbounded.

Comment by Githook User [ 01/Aug/16 ]

Author:

{u'username': u'mikety', u'name': u'Misha Tyulenev', u'email': u'misha@mongodb.com'}

Message: SERVER-23192 periodically reload ShardRegistry from the catalog
Branch: master
https://github.com/mongodb/mongo/commit/f32f51272b44f895875ab36fc0ae1553d11f1cc2

Comment by Githook User [ 25/Jul/16 ]

Author:

{u'username': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@mongodb.com'}

Message: SERVER-23192 Ensure we always stop monitoring replica sets that failed to be added as shards
Branch: master
https://github.com/mongodb/mongo/commit/cea106e6a3bec3b06ec8c8cab3c5c888d1af665a

Comment by Kaloian Manassiev [ 15/Jul/16 ]

This is correct. Thank you vlad.vintila@hootsuite.com for pointing out the incomplete title. We have updated the title to be more descriptive.

Comment by Vlad Vintila [ 15/Jul/16 ]

So just to reiterate here, mongos AND mongod have this issue, and they all need to be restarted. The title should be updated.

We've had all config servers down for more than 30s, and restarting mongos resulted in writes being accepted, but reads were still giving the error. Reads started working once we restarted ALL mongod servers(shards) of our affected cluster.

Comment by riccardo salzer [ 04/May/16 ]

today we had a connection problem between two datacenters and one shard, which had no config server in the same datacenter, went unavailable for read & write requests
mongod.log

2016-05-04T07:56:18.081+0000 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set mc-rs-cfgsrv-dev could be contacted.
2

mongos error message while insert

"errmsg" : "no progress was made executing batch write op in morecast_community.user after 5 rounds (0 ops completed in 6 rounds total)"

even when the connection came back after 30min, mongod was still of the opinion that the config servers are unavailable.
only manual interaction, restarting the mongd instances of the affected shard, could solve the problem. version is 3.2.5.
looking forward for this beeing fixed
cheers

Comment by Githook User [ 26/Apr/16 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-23192 startup_with_all_configs_down.js should restart the shards

While the config server is down, the shards' replica set monitor might
stop monitoring the config server and deem it permanently failed,
therefore to make the test robust, the mongod must be restarted.
Branch: master
https://github.com/mongodb/mongo/commit/1836409ef62456fb83361c2eb1a1e9189a5e3e60

Comment by Kaloian Manassiev [ 17/Mar/16 ]

Yes, this would be ideal, but just FYI - we won't be able to do it until we remove DBClientRS. Otherwise we will do duplicate monitoring.

Comment by Spencer Brody (Inactive) [ 16/Mar/16 ]

I feel like the long-term fix is to tie the lifetime of the replica set monitor to the lifetime of the Shard existing in the ShardRegistry. Something to consider for the ShardRegistry refactoring. FYI misha.tyulenev

Comment by Kaloian Manassiev [ 16/Mar/16 ]

One potential solution would be to extend the replica set monitor so it accepts a configurable upper bound on when to stop monitoring replica sets and special-case the CSRS replica set so we never give up on monitoring it.

Similar problem exists for the shard hosts. However since we periodically (every 10 seconds) refresh the ShardRegistry, these will eventually start being monitored again.

Generated at Thu Feb 08 04:02:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.