[SERVER-72532] CommandNotFound: no such command: 'shardVersion' Created: 05/Jan/23  Updated: 06/Nov/23  Resolved: 16/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.11, 5.0.4, 6.0.0, 6.2.0-rc4
Fix Version/s: 7.1.0-rc0, 4.4.26, 7.0.4

Type: Bug Priority: Major - P3
Reporter: dongyu si Assignee: Jada Lilleboe (Inactive)
Resolution: Fixed Votes: 0
Labels: neweng, sharding-nyc-subteam3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

os: centos 7 x86_64
mongo version: 4.4.14


Issue Links:
Backports
Depends
depends on SERVER-76004 Remove incorrect sharding tassert in ... Closed
Problem/Incident
is caused by SERVER-55412 Mirrored reads should propagate the s... Closed
Assigned Teams:
Sharding NYC
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0, v6.3, v6.0, v5.0, v4.4
Sprint: Service Arch 2023-02-20, Service Arch 2023-03-06, Sharding NYC 2023-04-03, Sharding NYC 2023-04-17, Sharding NYC 2023-05-01, Sharding NYC 2023-05-15, Sharding NYC 2023-05-29, Sharding NYC 2023-06-12, Sharding NYC 2023-06-26, Sharding NYC 2023-07-10, Sharding NYC 2023-07-24, Sharding NYC 2023-08-07, Sharding NYC 2023-08-21
Participants:

 Description   

Early this AM I deploy the mongod data servers in our sharded (3 shards, 3 nodes each, no arbiter) cluster. Since deployed our monitoring system has been detecting an uptick in user assertions. I have been searching for a cause, and after setting the loglevel to 1 and grabbing a snippet from the logs, it appears that this is the error:

{"t":{"$date":"2023-01-04T17:08:21.381+08:00"},"s":"D1", "c":"-",        "id":23074,   "ctx":"conn141332","msg":"User assertion","attr":{"error":"CommandNotFound: no such command: 'shardVersion'","file":"src/mongo/db/service_entry_point_common.cpp","line":1361}}
{"t":{"$date":"2023-01-04T17:08:21.381+08:00"},"s":"D1", "c":"COMMAND",  "id":21966,   "ctx":"conn141332","msg":"Assertion while executing command","attr":{"command":"shardVersion","db":"orderdb","error":"CommandNotFound: no such command: 'shardVersion'"}}
{"t":{"$date":"2023-01-04T17:08:21.381+08:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn141332","msg":"Slow query","attr":{"type":"command","ns":"orderdb.$cmd","command":"unrecognized","numYields":0,"ok":0,"errMsg":"no such command: 'shardVersion'","errName":"CommandNotFound","errCode":59,"locks":{},"protocol":"op_msg","durationMillis":0}}
{"t":{"$date":"2023-01-04T17:08:21.381+08:00"},"s":"D1", "c":"QUERY",    "id":22790,   "ctx":"conn141332","msg":"Received interrupt request for unknown op","attr":{"opId":3652609564}} 

This is happening on all 2 secondary shard servers, but not the primaries. The collections in these sharded dbs are still getting new documents, and replication appears to be working fine. I have not upgraded the mongos instances yet, those will be this weekend/next week to prevent impacts on production apps. Anyone know what might be causing this? Is this a bug for mongodb 4.4.x?

 

 

 

 

 



 Comments   
Comment by Githook User [ 06/Nov/23 ]

Author:

{'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object. (#16519)

GitOrigin-RevId: 3feee8d776027aa084d188ed79c780b87870e437
Branch: v4.4
https://github.com/mongodb/mongo/commit/5b5beb79c99fb2fb6eb20d3351750ca2b1b8558f

Comment by Githook User [ 05/Nov/23 ]

Author:

{'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object. (#16519)

GitOrigin-RevId: 8ea655357df35acb7ec48b180cd8b08587bb4715
Branch: v4.4
https://github.com/mongodb/mongo/commit/3feee8d776027aa084d188ed79c780b87870e437

Comment by Githook User [ 05/Nov/23 ]

Author:

{'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object. (#16519)

GitOrigin-RevId: 38dea79f43b4da48705b70d5711505ddb6aa23bd
Branch: v4.4
https://github.com/mongodb/mongo/commit/8ea655357df35acb7ec48b180cd8b08587bb4715

Comment by Githook User [ 04/Nov/23 ]

Author:

{'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object. (#16519)

GitOrigin-RevId: d10b185c027e9f2f76c07678a9713e638e333f02
Branch: v4.4
https://github.com/mongodb/mongo/commit/38dea79f43b4da48705b70d5711505ddb6aa23bd

Comment by Githook User [ 31/Oct/23 ]

Author:

{'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object. (#16519)

GitOrigin-RevId: b799367dd223cefb5e606714b1676d0b81756ee4
Branch: v4.4
https://github.com/mongodb/mongo/commit/d10b185c027e9f2f76c07678a9713e638e333f02

Comment by Githook User [ 30/Oct/23 ]

Author:

{'name': 'Nandini Bhartiya', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}

Message: SERVER-72532: Add shard & DB version field later in the BSON object.
Branch: v7.0
https://github.com/mongodb/mongo/commit/f3f979694f60427abd033fee112e92d7c7d69ead

Comment by Githook User [ 25/Jul/23 ]

Author:

{'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}

Message: SERVER-72532: add shard version field later in the BSON object

(cherry picked from commit 05f790d60bc7ce8d5b9b61bea8370b1c1f97837d)
Branch: v6.0
https://github.com/mongodb/mongo/commit/44ccddb01020f06989857365bf713f4d5cc3a286

Comment by Githook User [ 25/Jul/23 ]

Author:

{'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}

Message: SERVER-72532: add shard version field later in the BSON object

(cherry picked from commit 05f790d60bc7ce8d5b9b61bea8370b1c1f97837d)
Branch: v5.0
https://github.com/mongodb/mongo/commit/478b5317b58e1f17e14f5677ff128849c481eb2f

Comment by Githook User [ 23/Jun/23 ]

Author:

{'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}

Message: SERVER-72532: add shard version field later in the BSON object
Branch: master
https://github.com/mongodb/mongo/commit/05f790d60bc7ce8d5b9b61bea8370b1c1f97837d

Comment by Adi Zaimi [ 14/Apr/23 ]

Sergi identified a revert issue which triggers the tassert, and it will be handled in https://jira.mongodb.org/browse/SERVER-76004. Once that is fixed, we can continue with the fix here.

Comment by Adi Zaimi [ 09/Apr/23 ]

I can replicate the failure that Sam saw locally as well and it seems that we are hitting a tassert here (from Sam's run):
https://parsley.mongodb.com/resmoke/fc277ef96ae40f862b0e6721e73c4153/test/173779f2f5a0c87148a0de15d21674b4?bookmarks=0,10789,17408,17452,17494,17526,17661,20226,20337&shareLine=0

which is in https://github.com/10gen/mongo/blob/0239589385ef353ca1ee2a10036cd8c5c52d8599/src/mongo/db/s/collection_sharding_runtime.cpp#L148

    if (!supportNonVersionedOperations) {
        tassert(7032301,
                "For sharded collections getOwnershipFilter cannot be relied on without a valid "
                "shard version",
                !ShardVersion::isPlacementVersionIgnored(*optReceivedShardVersion) ||
                    !metadata->get().allowMigrations() || !metadata->get().isSharded());
    } 

Comment by dongyu si [ 06/Jan/23 ]

Hi, max.hirschhorn@mongodb.com, think you very much, I will upgrade the version after the bug fixed.

Comment by Max Hirschhorn [ 06/Jan/23 ]

Hi 335612970@qq.com, thank you the reporting this issue. The log messages you included and the additional details of how you are only seeing "CommandNotFound: no such command: 'shardVersion'" on secondary members of the replica set shards were helpful for us to identify the issue quickly.

We have identified the issue as being caused by the changes from SERVER-55412. The primary of a replica set mirrors certain operations to its secondary members to keep the caches on the secondaries warm in the event one of the secondaries must take over as primary. Update commands intend to mirror a find command to the secondary based on the filter from the update. However the changes from SERVER-55412 are causing the mirrored request to be sent as a non-existent "shardVersion" command. This is due to how the name of the command being implied by the first field in the BSON document.

https://www.mongodb.com/docs/manual/replication/#std-label-mirrored-reads

if (const auto& shardVersion = _commandObj.getField("shardVersion");
    !shardVersion.eoo()) {
    bob->append(shardVersion);
}
 
bob->append("find", _commandObj["update"].String());
extractQueryDetails(_updateOpObj, bob);
bob->append("batchSize", 1);
bob->append("singleBatch", true);

The team is working on a patch and will make the appropriate changes across the affected branches. If in the meantime you are wanting to avoid triggering these errors in your sharded cluster then you may choose to temporarily disable the mirror reads feature entirely by adding the following server parameter to your mongod command line. Note that the default sampling rate is 1%.

--setParameter mirrorReads='{samplingRate: 0}'

Thanks,
Max

Comment by dongyu si [ 06/Jan/23 ]

This is happening on all 2 secondary shard servers, but not the primaries. The collections in these sharded dbs are still getting new documents, and replication appears to be working fine. Anyone know what might be causing this? Is this a bug for mongodb 4.4.x? All mongos, configsvr, shardsvr  are 4.4.14 version.

Generated at Thu Feb 08 06:22:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.