[SERVER-40816] Can't enable 3.6 features on Shard and can't restart mongos after it Created: 24/Apr/19  Updated: 15/Nov/19  Resolved: 25/Apr/19

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.6.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sarbamangal Choudhury Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I upgraded Mongo Shard from 3.4 to 3.6 and trying to enable this:

mongos> db.adminCommand( \{ setFeatureCompatibilityVersion: "3.6" } );
{
 "ok" : 0,
 "errmsg" : "Could not find host matching read preference \{ mode: \"primary\" } for set shard2",
 "code" : 133,
 "codeName" : "FailedToSatisfyReadPreference"
}
mongos>

 

Now FeatureCompatibilityVersion: "3.6" is set on Config and shard. 

configReplSet:SECONDARY> db.adminCommand( \{ getParameter: 1, featureCompatibilityVersion: 1});
{
 "featureCompatibilityVersion" : {
 "version" : "3.4",
 "targetVersion" : "3.6"
 },
 "ok" : 1,
 "$gleStats" : {
 "lastOpTime" : Timestamp(0, 0),
 "electionId" : ObjectId("000000000000000000000000")
 }
}
configReplSet:SECONDARY>

shard1:SECONDARY> db.adminCommand( \{ getParameter: 1, featureCompatibilityVersion: 1})
{
 "featureCompatibilityVersion" : {
 "version" : "3.6"
 },
 "ok" : 1,
 "$gleStats" : {
 "lastOpTime" : Timestamp(0, 0),
 "electionId" : ObjectId("000000000000000000000000")
 },
 "$configServerState" : {
 "opTime" : {
 "ts" : Timestamp(1556145038, 2),
 "t" : NumberLong(7)
 }
 }
}
shard1:SECONDARY>

 

Now try to bounce mongos server and face this error:

tail -f /vol/logs/mongos.log
2019-04-24T22:42:20.951+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:21.952+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:22.952+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:23.952+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:24.952+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:25.952+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:26.953+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:27.953+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:28.953+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:29.953+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:30.953+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:31.954+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.
2019-04-24T22:42:32.954+0000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.

 



 Comments   
Comment by Danny Hatcher (Inactive) [ 15/Nov/19 ]

I'm glad you were able to resolve it!

Comment by Timur Gilauri [ 15/Nov/19 ]

I have solved my problem by makin a full cluster restart, starting from mongos down to config servers and back. On the machines with the mongos instances I didn't just stoped instances but reboot machines. And it worked for me. Guess there was some problems with syncing operations.

Comment by Danny Hatcher (Inactive) [ 13/Nov/19 ]

Please try upgrading to a later version of 3.6 instead (3.6.15 is the latest at the time of this comment). If you followed all of our instructions for upgrading to 3.6 for a sharded cluster, are on 3.6.15, and you are still seeing these issues, please open a new SERVER ticket.

Comment by Timur Gilauri [ 13/Nov/19 ]

What if all shards replica set primary hosts are available? 

In my case, I faced this issue during shared cluster update tfrom 3.4 to 3.6.8. I upgraded config servers replica set, both shards and then tryed to upgrade mongos instances. After upgrade and restart instance on one of the routers it stalled  on systemctl start mongos and I can see in logs that mongos has started fine and established connections to config servers, but then just write the same error:

2019-11-13T14:40:26.508+1000 I SHARDING [mongosMain] Waiting for signing keys, sleeping for 1s and trying again.

and after some time timout exceeded. 

The config servers replica set  and shards servers replica sets are fine. rs.ststus() is fine. Bind_ip is set explicitly. No mistakes in configDB  property in replica set name and addresses. 

Comment by Danny Hatcher (Inactive) [ 25/Apr/19 ]

Hello,

The error in your setFeatureCompabilityVersion command states that shard2 didn't have a Primary at the time the command was run:

 "errmsg" : "Could not find host matching read preference \{ mode: \"primary\" } for set shard2",

As stated in our Upgrade Documentation, all shards must be replica sets at the time of the upgrade. This also means that shards must be healthy enough to elect a Primary. As the command could not find a Primary for "shard2", it had not yet finished. That can be seen by the existence of the targetVersion in the config server secondary which implies that the cluster has not fully converted to FCV 3.6.

Please ensure that each of your shards has a Primary node that can except writes and run the following on a mongos again:

db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

In regards to the key messages in the mongos, it is likely that you are experiencing SERVER-40136. This is fixed in 3.6.13 which will be our next update in the 3.6 series.

As there does not appear to be a bug in the server that caused this issue, I will now close this ticket. If you need further assistance troubleshooting, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Thanks,

Danny

Generated at Thu Feb 08 04:56:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.