[SERVER-47568] No keys found for HMAC in RECOVERING state Created: 15/Apr/20  Updated: 29/Oct/23  Resolved: 04/May/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.17, 4.2.5, 4.0.18
Fix Version/s: 4.2.13, 4.4.5, 4.0.24, 5.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Misha Tyulenev Assignee: Jack Mulrow
Resolution: Fixed Votes: 4
Labels: sharding-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-54281 VectorClock's configTime can become g... Closed
Problem/Incident
Related
is related to SERVER-55371 4.2.13 removed rs.initiate() return o... Closed
is related to SERVER-56471 Donor's listDatabases response missin... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2, v4.0
Sprint: Sharding 2020-09-07, Sharding 2021-01-11, Sharding 2021-01-25, Sharding 2021-02-22, Sharding 2021-03-08, Sharding 2021-03-22, Sharding 2021-04-05, Sharding 2021-04-19, Sharding 2021-05-03, Sharding 2021-05-17
Participants:
Case:
Linked BF Score: 142

 Description   

Cluster time validation reads signing keys from cache , and if there are no keys the cache is refreshed by reading admin.system.keys collection. When the server is in the RECOVERING state it can not read from any collection including admin.system.keys collection. Hence any request, in particular isMaster from a monitoring agent returns meesages similar to

KeyNotFound) Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1585658632, 12) } with id: 6769620021938225153



 Comments   
Comment by Githook User [ 04/May/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Disable clusterTime gossiping for nodes in unreadable states
Branch: master
https://github.com/mongodb/mongo/commit/a8e91314ab13d431e7e0f2ce10c5e5aa63c0c58d

Comment by Githook User [ 11/Mar/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Allow multiple error codes in read_concern_uninitiated_set.js with mixed bin versions
Branch: v4.4
https://github.com/mongodb/mongo/commit/774adf3e6020fff627436d1be3f5d2152cc74ecf

Comment by Githook User [ 09/Mar/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Temporarily relax read_concern_uninitiated_set.js with mixed bin versions
Branch: master
https://github.com/mongodb/mongo/commit/7db8fdb7ff39b14a0a09996e67968a5a715f754b

Comment by Githook User [ 04/Mar/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Disable clusterTime gossiping for nodes in unreadable states

(cherry picked from commit 024b130c5e66bafd99cf7f899cdef8d23284ef81)
Branch: v4.0
https://github.com/mongodb/mongo/commit/9bfad127acfbfa46f2782a92f3ea698519ac6f08

Comment by Githook User [ 04/Mar/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Disable clusterTime gossiping for nodes in unreadable states

(cherry picked from commit 024b130c5e66bafd99cf7f899cdef8d23284ef81)
Branch: v4.2
https://github.com/mongodb/mongo/commit/82dd40f60c55dae12426c08fd7150d79a0e28e23

Comment by Githook User [ 03/Mar/21 ]

Author:

{'name': 'Jack Mulrow', 'email': 'jack.mulrow@mongodb.com', 'username': 'jsmulrow'}

Message: SERVER-47568 Disable clusterTime gossiping for nodes in unreadable states
Branch: v4.4
https://github.com/mongodb/mongo/commit/024b130c5e66bafd99cf7f899cdef8d23284ef81

Comment by Githook User [ 05/Aug/20 ]

Author:

{'name': 'Misha Tyulenev', 'email': 'misha.tyulenev@mongodb.com'}

Message: Revert "SERVER-47568 disable clusterTime gossiping for nodes not in readable state"

This reverts commit 3c9d3a05755057791a045214b2580e7074291ef6.
Branch: v4.4
https://github.com/mongodb/mongo/commit/9e4e06d0a7db9d2c81ffb84f91708e2b2134dbfa

Comment by Githook User [ 04/Aug/20 ]

Author:

{'name': 'Misha Tyulenev', 'email': 'misha.tyulenev@mongodb.com'}

Message: SERVER-47568 disable clusterTime gossiping for nodes not in readable state
Branch: v4.4
https://github.com/mongodb/mongo/commit/3c9d3a05755057791a045214b2580e7074291ef6

Comment by Misha Tyulenev [ 29/Jul/20 ]

Note: after the fix its possible to get the InvalidOptions error when processing atClusterTime or afterClusterTime on a non readable node due to the check for atClusterTime or afterClusterTime to be greater than the current node's clusterTime.

Comment by Kostiantyn Dobarskyi [ 27/Jul/20 ]

I see exactly the same exceptions reported by Mongo driver connected to an Atlas replica set (version 4.0.19) while cluster maintenance is in progress.

Comment by Misha Tyulenev [ 04/Jun/20 ]

Implemented a fix, but could not reproduce the scenario in a testcase. Back to the HELP ticket scenario to find out the exact scenario.

Generated at Thu Feb 08 05:14:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.