[SERVER-66546] Failed to refresh key cache - AuthenticationFailed Created: 18/May/22 Updated: 11/Jul/22 Resolved: 11/Jul/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 5.0.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Pankaj Dahiya | Assignee: | Chris Kelly |
| Resolution: | Done | Votes: | 0 |
| Labels: | confirmed_bug, kubernetes, sharded-cluster | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | 1. helm repo add bitnami https://charts.bitnami.com/bitnami 2. helm install my-release bitnami/mongodb-sharded 3. Let the cluster keep running for a few days and it will crash itself!!
The same type of error was reported last year by some users:
I logged the issue with bitnami too and they asked me to contact mongoDB team |
| Participants: |
| Description |
|
I am launching MongoDb sharded cluster using Bitnami Helm charts While installing at day zero, all things working fine and we are able to push data into MongoDB with sharding enabled. But after 2-3 days, suddenly MongoDB router starts misbehaving and goes into crassLoopBack state with below logs: k logs common-mongodb-mongos-0 --tail=100 {"t":\{"$date":"2022-05-18T07:02:36.122+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ShardRegistry-0","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}} {"t":\{"$date":"2022-05-18T07:02:36.122+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}} {"t":\{"$date":"2022-05-18T07:02:36.124+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"73346dac-26d5-4c62-a188-e89375c4defd\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 630, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356124), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"ecc9336a-ac8b-402a-bab8-4327d885ed54\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:36.124+00:00"},"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:36.124+00:00"},"s":"I", "c":"SHARDING", "id":22846, "ctx":"ShardRegistry::reload","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:36.125+00:00"},"s":"I", "c":"-", "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":"AuthenticationFailed: Authentication failed.","replicaSet":"common-mongodb-configsvr","response":"{}"}} {"t":\{"$date":"2022-05-18T07:02:36.126+00:00"},"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."},"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}} {"t":\{"$date":"2022-05-18T07:02:36.126+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"ba1d5a5e-e30b-442c-8540-2da48e69af9c\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"73346dac-26d5-4c62-a188-e89375c4defd\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 630, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356124), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:36.130+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:36.130+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:36.130+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:36.191+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"eb385d11-c98e-4288-b528-eac2132219b3\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 725, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356191), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"ba1d5a5e-e30b-442c-8540-2da48e69af9c\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:36.191+00:00"},"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:36.191+00:00"},"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:36.195+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:36.733+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:37.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:38.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:38.733+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4200}} {"t":\{"$date":"2022-05-18T07:02:38.741+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:39.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:40.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:40.745+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:41.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:42.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:42.749+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:42.941+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4400}} {"t":\{"$date":"2022-05-18T07:02:43.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:44.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:44.753+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:45.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.120+00:00"},"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."},"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}} {"t":\{"$date":"2022-05-18T07:02:46.121+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"e19c562b-2ccf-4e1e-9a5e-fdd25209d1a5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"97bc021f-7ada-4fa5-8e52-841fb4627c0f\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 786, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366119), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:46.121+00:00"},"s":"I", "c":"-", "id":4333227, "ctx":"ShardRegistry-0","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","replicaSet":"common-mongodb-configsvr"}} {"t":\{"$date":"2022-05-18T07:02:46.121+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ShardRegistry-0","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}} {"t":\{"$date":"2022-05-18T07:02:46.121+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}} {"t":\{"$date":"2022-05-18T07:02:46.123+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"78de2428-eb48-4a26-95da-5962fb0f1240\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 589, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366123), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"e19c562b-2ccf-4e1e-9a5e-fdd25209d1a5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:46.123+00:00"},"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.123+00:00"},"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.124+00:00"},"s":"I", "c":"-", "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":"AuthenticationFailed: Authentication failed.","replicaSet":"common-mongodb-configsvr","response":"{}"}} {"t":\{"$date":"2022-05-18T07:02:46.124+00:00"},"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."},"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}} {"t":\{"$date":"2022-05-18T07:02:46.124+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"5d7587f0-ce54-457f-b9ee-fc4c1e3702d5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"78de2428-eb48-4a26-95da-5962fb0f1240\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 589, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366123), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:46.126+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:46.126+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:46.126+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:46.191+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"9d8298fb-21e7-4944-abaf-543767bd0765\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion: { processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }, roundTripTime: 717, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366191), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"5d7587f0-ce54-457f-b9ee-fc4c1e3702d5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}} {"t":\{"$date":"2022-05-18T07:02:46.191+00:00"},"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.191+00:00"},"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:46.195+00:00"},"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}} {"t":\{"$date":"2022-05-18T07:02:46.758+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:47.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:47.348+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4600}} {"t":\{"$date":"2022-05-18T07:02:48.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:48.771+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:49.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:50.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:50.775+00:00"},"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error": {"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}}} {"t":\{"$date":"2022-05-18T07:02:51.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} {"t":\{"$date":"2022-05-18T07:02:51.954+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4800}} {"t":\{"$date":"2022-05-18T07:02:52.117+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}} |
| Comments |
| Comment by Chris Kelly [ 11/Jul/22 ] |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. |
| Comment by Chris Kelly [ 01/Jul/22 ] |
|
We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information? |
| Comment by Chris Kelly [ 31/May/22 ] |
|
Hi Pankaj, When looking at If you're able, it'd be super helpful to try reproducing this issue again and collecting the complete mongo logs for us. You can upload these directly to the ticket, but if you're having issues I've also gone ahead and created an upload link as well. Would you please archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Thanks, Christopher |
| Comment by Pankaj Dahiya [ 25/May/22 ] |
|
Hi Chris, I do not have more logs, your Jira ticket has a limit of words, so I have added that many logs here. Then after 5 days, we do not have logs. We cleaned MongoDB by deleting all the data directories and installed MongoDB again! If you have commented on the same day or even on subsequent days, I would have shared logs with you! But now, We have nothing to share! |
| Comment by Chris Kelly [ 23/May/22 ] |
|
Hi Pankaj, In order to look into this further, we will need you to archive (tar or zip) the following and upload them to the ticket:
Christopher |