[SERVER-46702] resmoke.py-based sharding multiversion tests start clusters using latest mongos and fail Created: 07/Mar/20  Updated: 29/Oct/23  Resolved: 10/Mar/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Repl 2020-03-23
Participants:
Linked BF Score: 49

 Description   

The mongos binary should be upgraded last (and downgraded first) according to our documentation, so we should always be using "last-stable" when --mixedBinVersions has been specified. Not doing so prevents the mongos from communicating with the shards.

PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongod --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'sharding': {'migration': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --configsvr --replSet=config-rs --storageEngine=wiredTiger --oplogSize=511 --dbpath=/data/db/job0/resmoke/config/node0 --port=20000 --journal --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongod --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'sharding': {'migration': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --configsvr --replSet=config-rs --storageEngine=wiredTiger --oplogSize=511 --dbpath=/data/db/job0/resmoke/config/node1 --port=20001 --journal --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongod --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'sharding': {'migration': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter orphanCleanupDelaySecs=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --oplogSize=1024 --shardsvr --replSet=shard-rs0 --dbpath=/data/db/job0/resmoke/shard0/node0 --port=20002 --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... mongod-4.2 --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter orphanCleanupDelaySecs=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --oplogSize=1024 --shardsvr --replSet=shard-rs0 --dbpath=/data/db/job0/resmoke/shard0/node1 --port=20003 --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... mongod-4.2 --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter orphanCleanupDelaySecs=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --oplogSize=1024 --shardsvr --replSet=shard-rs1 --dbpath=/data/db/job0/resmoke/shard1/node0 --port=20004 --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongod --setParameter enableTestCommands=1 --setParameter migrationLockAcquisitionMaxWaitMS=30000 --setParameter logComponentVerbosity={'replication': {'election': 4, 'heartbeats': 2, 'initialSync': 2, 'rollback': 2}, 'sharding': {'migration': 2}, 'storage': {'recovery': 2}, 'transaction': 4} --setParameter orphanCleanupDelaySecs=1 --setParameter disableLogicalSessionCacheRefresh=true --setParameter transactionLifetimeLimitSeconds=86400 --setParameter maxIndexBuildDrainBatchSize=10 --setParameter writePeriodicNoops=false --setParameter waitForStepDownOnNonCommandShutdown=false --oplogSize=1024 --shardsvr --replSet=shard-rs1 --dbpath=/data/db/job0/resmoke/shard1/node1 --port=20005 --enableMajorityReadConcern=True --wiredTigerCacheSizeGB=1
PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongos --setParameter enableTestCommands=1 --setParameter logComponentVerbosity={'transaction': 3} --configdb=config-rs/localhost:20000,localhost:20001 --port=20006
PATH=... /data/mci/b4355878e0cb15daa440ed2ec90b1ac9/src/dist-test/bin/mongos --setParameter enableTestCommands=1 --setParameter logComponentVerbosity={'transaction': 3} --configdb=config-rs/localhost:20000,localhost:20001 --port=20007
...
[ShardedClusterFixture:job0:mongos0] | 2020-03-06T18:13:54.403+0000 W  NETWORK  23741   [ReplicaSetMonitor-TaskExecutor] "remote host has incompatible wire version: {validateStatus}","attr":{"validateStatus":{"code":188,"codeName":"IncompatibleServerVersion","errmsg":"Server min and max wire version (8,8) is incompatible with client min wire version (9,9).You (client) are attempting to connect to a node (server) with a binary version with which you (client) no longer accept connections. Please upgrade the server’s binary version."}}
[ShardedClusterFixture:job0:mongos0] | 2020-03-06T18:13:54.403+0000 I  -        4333222 [ReplicaSetMonitor-TaskExecutor] "RSM {setName} received failed isMaster for server {host}: {status} ({latency}): {bson}","attr":{"host":"localhost:20003","status":"IncompatibleServerVersion: remote host has incompatible wire version: Server min and max wire version (8,8) is incompatible with client min wire version (9,9).You (client) are attempting to connect to a node (server) with a binary version with which you (client) no longer accept connections. Please upgrade the server’s binary version.","latencyNanos":1419000,"setName":"shard-rs0","bson":"{}"}
[ShardedClusterFixture:job0:mongos0] | 2020-03-06T18:13:54.403+0000 I  NETWORK  4333213 [ReplicaSetMonitor-TaskExecutor] "RSM {setName} Topology Change: {topologyDescription}","attr":{"setName":"shard-rs0","topologyDescription":"{ id: \"4827e4c6-549b-4a50-8211-de26cd99819b\", topologyType: \"Unknown\", servers: { localhost:20002: { address: \"localhost:20002\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, streamable: false, poolResetCounter: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: [], arbiters: [], passives: [] }, localhost:20003: { address: \"localhost:20003\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, streamable: false, poolResetCounter: 1, lastUpdateTime: new Date(-9223372036854775808), hosts: [], arbiters: [], passives: [] } }, compatible: true }"}

https://logkeeper.mongodb.org/lobster/build/03ec4e4f0a9dacddd1d8589cd13cdf97/all#bookmarks=0%2C2154%2C2155%2C2156%2C33940&f~=101~PATH%3D&l=1



 Comments   
Comment by Githook User [ 10/Mar/20 ]

Author:

{'username': 'jasonjhchan', 'name': 'Jason Chan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-46702 Use last-stable mongos version in sharding multiversion passthroughs

(cherry picked from commit e62d66e36b3c6b2322b8dd4d6d7dc80a8e7156c8)
Branch: v4.4
https://github.com/mongodb/mongo/commit/3ceb5fa7807568709c3773b755c2e8d3f299c030

Comment by Githook User [ 10/Mar/20 ]

Author:

{'name': 'Jason Chan', 'username': 'jasonjhchan', 'email': 'jason.chan@10gen.com'}

Message: SERVER-46702 Use last-stable mongos version in sharding multiversion passthroughs
Branch: master
https://github.com/mongodb/mongo/commit/e62d66e36b3c6b2322b8dd4d6d7dc80a8e7156c8

Comment by Max Hirschhorn [ 07/Mar/20 ]

Note that I assigned this ticket directly to Jason Chan at his own request.

Generated at Thu Feb 08 05:12:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.