[SERVER-11982] connection error when doing sharded cluster upgrade following recommended procedure Created: 05/Dec/13  Updated: 10/Mar/14  Resolved: 11/Feb/14

Status: Closed
Project: Core Server
Component/s: Admin, Sharding
Affects Version/s: 2.5.4
Fix Version/s: 2.6.0-rc0

Type: Bug Priority: Major - P3
Reporter: Adinoyi Omuya Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: 26qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on DOCS-2659 Document v2.6 upgrade order Closed
Duplicate
is duplicated by SERVER-12234 MongoS connections that persist acros... Closed
Related
related to DOCS-2682 doc 2.6 upgrade of sharded cluster Closed
is related to SERVER-4887 sharded queries on system.indexes hit... Closed
Operating System: ALL
Participants:

 Description   

The recommended process for doing a sharded cluster version upgrade is:

  • Upgrade mongos (with --upgrade)
  • Upgrade shards
  • Upgrade config servers

However, when running this for a 2.4 to 2.5 version upgrade, the following error is thrown when a client (e.g. a mongo process) attempts to run the authSchemaUpgradeStep command on the admin database.

{
	"ok" : 0,
	"errmsg" : "SyncClusterConnection::update prepare failed:  localhost.mongodb.com:29001:10276 DBClientBase::findN: transport error: localhost.mongodb.com:29001 ns: admin.$cmd query: { resetError: 1 } localhost.mongodb.com:29002:10276 DBClientBase::findN: transport error: localhost.mongodb.com:29002 ns: admin.$cmd query: { resetError: 1 }",
	"code" : 8005
}

However if the order is changed a bit:

  • Upgrade shards
  • Upgrade config servers
  • Upgrade mongos (with --upgrade)

There are no issues upgrading in this order.



 Comments   
Comment by Randolph Tan [ 11/Feb/14 ]

Note: if you follow the recommended procedure, you need to call connPoolSync against the mongos before running authSchemaUpgrade

Comment by Githook User [ 11/Feb/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-11982 connection error when doing sharded cluster upgrade following recommended procedure

Make connPoolSync clear all global connection pools
Branch: master
https://github.com/mongodb/mongo/commit/8a2536ddf2933e90c41ef5567aa6fbb0029547e9

Comment by Githook User [ 11/Feb/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-11982 connection error when doing sharded cluster upgrade following recommended procedure
Branch: master
https://github.com/mongodb/mongo/commit/228c5a2e27cdaf35e1ac7e0a097d8172f7f4a387

Comment by Randolph Tan [ 04/Feb/14 ]

To workaround this issue, the upgrade procedure should be done as follows:

1. Upgrade config server binaries.
2. Upgrade mongos with --upgrade
3. Upgrade shards
4. Run connPoolSync on mongos at step 2.
5. Run authSchemaUpgradeStep on mongos at step 2.

Comment by Valeri Karpov [ 12/Dec/13 ]

I can confirm that I've had a similar issue

Generated at Thu Feb 08 03:27:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.