ISSUE DESCRIPTION AND IMPACT

The bug causes a failure of the thread that creates new Hash-based Message Authentication Code (HMAC) signing keys every 90 days.

New keys are generated when the Config Server Replica Set (CSRS) fails over. So, if a failover does not happen on the CSRS for 90 days, operations across the sharded cluster will start to fail and will not succeed again until the CSRS fails over.

DIAGNOSIS AND AFFECTED VERSIONS

MongoDB 4.2.2 to 4.2.11 and 4.4.0 to 4.4.2 are affected. The bug may exist in previous versions but mechanisms other than failover cause the CSRS primary to re-generate the HMAC keys successfully in those versions.

To check the expiration date of the HMAC keys, use a mongo shell to connect to a mongos node, or the CSRS primary, authenticate as a user with admin privilege and run the following command to check the expiration date for the HMAC signing keys. The cluster will experience this issue when all the HMAC signing keys expire.

db.getSiblingDB("admin").system.keys.find().map(k => { return { _id: k._id, purpose: k.purpose, expiresAt: new Date(k.expiresAt.getTime()*1000) }})

To perform this check the database user must have permissions to query the admin.system.keys collection. To grant these permissions, create a new role with the find action on the admin.system.keys collection and grant this role to an admin user with the following commands, replacing ADMIN with the username:

use admin;

db.createRole({
  role: "query_keys",
  privileges: [
     { resource: { db: "admin", collection: "system.keys"}, actions: [ "find" ] },
  ],
  roles: [  ]
});

db.grantRolesToUser("ADMIN", ["query_keys"])

REMEDIATION AND WORKAROUNDS

The fix is included in the 3.6.22, 4.0.22, 4.2.12 and 4.4.3 production releases and later. To prevent the issue before upgrading to a fixed release, step down the CSRS primary to initiate a failover before the 90 days limit is reached.