[SERVER-72532] CommandNotFound: no such command: 'shardVersion' Created: 05/Jan/23 Updated: 06/Nov/23 Resolved: 16/Aug/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.4.11, 5.0.4, 6.0.0, 6.2.0-rc4 |
| Fix Version/s: | 7.1.0-rc0, 4.4.26, 7.0.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | dongyu si | Assignee: | Jada Lilleboe (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | neweng, sharding-nyc-subteam3 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
os: centos 7 x86_64 |
||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Sharding NYC
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Backport Requested: |
v7.0, v6.3, v6.0, v5.0, v4.4
|
||||||||||||||||||||
| Sprint: | Service Arch 2023-02-20, Service Arch 2023-03-06, Sharding NYC 2023-04-03, Sharding NYC 2023-04-17, Sharding NYC 2023-05-01, Sharding NYC 2023-05-15, Sharding NYC 2023-05-29, Sharding NYC 2023-06-12, Sharding NYC 2023-06-26, Sharding NYC 2023-07-10, Sharding NYC 2023-07-24, Sharding NYC 2023-08-07, Sharding NYC 2023-08-21 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Early this AM I deploy the mongod data servers in our sharded (3 shards, 3 nodes each, no arbiter) cluster. Since deployed our monitoring system has been detecting an uptick in user assertions. I have been searching for a cause, and after setting the loglevel to 1 and grabbing a snippet from the logs, it appears that this is the error:
This is happening on all 2 secondary shard servers, but not the primaries. The collections in these sharded dbs are still getting new documents, and replication appears to be working fine. I have not upgraded the mongos instances yet, those will be this weekend/next week to prevent impacts on production apps. Anyone know what might be causing this? Is this a bug for mongodb 4.4.x?
|
| Comments |
| Comment by Githook User [ 06/Nov/23 ] | ||||||||||
|
Author: {'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}Message: GitOrigin-RevId: 3feee8d776027aa084d188ed79c780b87870e437 | ||||||||||
| Comment by Githook User [ 05/Nov/23 ] | ||||||||||
|
Author: {'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}Message: GitOrigin-RevId: 8ea655357df35acb7ec48b180cd8b08587bb4715 | ||||||||||
| Comment by Githook User [ 05/Nov/23 ] | ||||||||||
|
Author: {'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}Message: GitOrigin-RevId: 38dea79f43b4da48705b70d5711505ddb6aa23bd | ||||||||||
| Comment by Githook User [ 04/Nov/23 ] | ||||||||||
|
Author: {'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}Message: GitOrigin-RevId: d10b185c027e9f2f76c07678a9713e638e333f02 | ||||||||||
| Comment by Githook User [ 31/Oct/23 ] | ||||||||||
|
Author: {'name': 'nandinibhartiyaMDB', 'email': '104035932+nandinibhartiyaMDB@users.noreply.github.com', 'username': 'nandinibhartiyaMDB'}Message: GitOrigin-RevId: b799367dd223cefb5e606714b1676d0b81756ee4 | ||||||||||
| Comment by Githook User [ 30/Oct/23 ] | ||||||||||
|
Author: {'name': 'Nandini Bhartiya', 'email': 'nandini.bhartiya@mongodb.com', 'username': 'nandinibhartiyaMDB'}Message: | ||||||||||
| Comment by Githook User [ 25/Jul/23 ] | ||||||||||
|
Author: {'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}Message: (cherry picked from commit 05f790d60bc7ce8d5b9b61bea8370b1c1f97837d) | ||||||||||
| Comment by Githook User [ 25/Jul/23 ] | ||||||||||
|
Author: {'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}Message: (cherry picked from commit 05f790d60bc7ce8d5b9b61bea8370b1c1f97837d) | ||||||||||
| Comment by Githook User [ 23/Jun/23 ] | ||||||||||
|
Author: {'name': 'Jada Lilleboe', 'email': 'jada.lilleboe@mongodb.com', 'username': 'jadalilleboe'}Message: | ||||||||||
| Comment by Adi Zaimi [ 14/Apr/23 ] | ||||||||||
|
Sergi identified a revert issue which triggers the tassert, and it will be handled in https://jira.mongodb.org/browse/SERVER-76004. Once that is fixed, we can continue with the fix here. | ||||||||||
| Comment by Adi Zaimi [ 09/Apr/23 ] | ||||||||||
|
I can replicate the failure that Sam saw locally as well and it seems that we are hitting a tassert here (from Sam's run):
| ||||||||||
| Comment by dongyu si [ 06/Jan/23 ] | ||||||||||
|
Hi, max.hirschhorn@mongodb.com, think you very much, I will upgrade the version after the bug fixed. | ||||||||||
| Comment by Max Hirschhorn [ 06/Jan/23 ] | ||||||||||
|
Hi 335612970@qq.com, thank you the reporting this issue. The log messages you included and the additional details of how you are only seeing "CommandNotFound: no such command: 'shardVersion'" on secondary members of the replica set shards were helpful for us to identify the issue quickly. We have identified the issue as being caused by the changes from https://www.mongodb.com/docs/manual/replication/#std-label-mirrored-reads
The team is working on a patch and will make the appropriate changes across the affected branches. If in the meantime you are wanting to avoid triggering these errors in your sharded cluster then you may choose to temporarily disable the mirror reads feature entirely by adding the following server parameter to your mongod command line. Note that the default sampling rate is 1%.
Thanks, | ||||||||||
| Comment by dongyu si [ 06/Jan/23 ] | ||||||||||
|
This is happening on all 2 secondary shard servers, but not the primaries. The collections in these sharded dbs are still getting new documents, and replication appears to be working fine. Anyone know what might be causing this? Is this a bug for mongodb 4.4.x? All mongos, configsvr, shardsvr are 4.4.14 version. |