Loading...

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 5.0.8
Component/s: None
Labels:

Operating System:
ALL
Steps To Reproduce:

Hide

1. helm repo add bitnami https://charts.bitnami.com/bitnami

2. helm install my-release bitnami/mongodb-sharded

3. Let the cluster keep running for a few days and it will crash itself!!

The same type of error was reported last year by some users:
https://jira.mongodb.org/browse/SERVER-53847

I logged the issue with bitnami too and they asked me to contact mongoDB team
https://github.com/bitnami/charts/issues/10288

Show
1. helm repo add bitnami https://charts.bitnami.com/bitnami 2. helm install my-release bitnami/mongodb-sharded 3. Let the cluster keep running for a few days and it will crash itself!! The same type of error was reported last year by some users: https://jira.mongodb.org/browse/SERVER-53847 I logged the issue with bitnami too and they asked me to contact mongoDB team https://github.com/bitnami/charts/issues/10288

I am launching MongoDb sharded cluster using Bitnami Helm charts
https://github.com/bitnami/charts/tree/master/bitnami/mongodb-sharded

While installing at day zero, all things working fine and we are able to push data into MongoDB with sharding enabled. But after 2-3 days, suddenly MongoDB router starts misbehaving and goes into crassLoopBack state with below logs:

k logs common-mongodb-mongos-0 --tail=100

{"t":\{"$date":"2022-05-18T07:02:36.122+00:00"}

,"s":"I", "c":"-", "id":4333218, "ctx":"ShardRegistry-0","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}}

{"t":\{"$date":"2022-05-18T07:02:36.122+00:00"}

,"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}}

{"t":\{"$date":"2022-05-18T07:02:36.124+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"73346dac-26d5-4c62-a188-e89375c4defd\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 630, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356124), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"ecc9336a-ac8b-402a-bab8-4327d885ed54\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:36.124+00:00"}

,"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:36.124+00:00"}

,"s":"I", "c":"SHARDING", "id":22846, "ctx":"ShardRegistry::reload","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:36.125+00:00"}

,"s":"I", "c":"-", "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":"AuthenticationFailed: Authentication failed.","replicaSet":"common-mongodb-configsvr","response":"{}"}}

{"t":\{"$date":"2022-05-18T07:02:36.126+00:00"}

,"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."}

,"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}}

{"t":\{"$date":"2022-05-18T07:02:36.126+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"ba1d5a5e-e30b-442c-8540-2da48e69af9c\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"73346dac-26d5-4c62-a188-e89375c4defd\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 630, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356124), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:36.130+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:36.130+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:36.130+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:36.191+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"eb385d11-c98e-4288-b528-eac2132219b3\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 725, lastWriteDate: new Date(1652857355000), opTime: { ts: Timestamp(1652857355, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857356191), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"ba1d5a5e-e30b-442c-8540-2da48e69af9c\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:36.191+00:00"}

,"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:36.191+00:00"}

,"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:36.195+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:36.733+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:37.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:38.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:38.733+00:00"}

,"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4200}}

{"t":\{"$date":"2022-05-18T07:02:38.741+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:39.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:40.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:40.745+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:41.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:42.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:42.749+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:42.941+00:00"}

,"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4400}}

{"t":\{"$date":"2022-05-18T07:02:43.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:44.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:44.753+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:45.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.120+00:00"}

,"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."}

,"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}}

{"t":\{"$date":"2022-05-18T07:02:46.121+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"e19c562b-2ccf-4e1e-9a5e-fdd25209d1a5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"97bc021f-7ada-4fa5-8e52-841fb4627c0f\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 786, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366119), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:46.121+00:00"}

,"s":"I", "c":"-", "id":4333227, "ctx":"ShardRegistry-0","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","replicaSet":"common-mongodb-configsvr"}}

{"t":\{"$date":"2022-05-18T07:02:46.121+00:00"}

,"s":"I", "c":"-", "id":4333218, "ctx":"ShardRegistry-0","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}}

{"t":\{"$date":"2022-05-18T07:02:46.121+00:00"}

,"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","delayMillis":0}}

{"t":\{"$date":"2022-05-18T07:02:46.123+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"78de2428-eb48-4a26-95da-5962fb0f1240\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 589, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366123), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"e19c562b-2ccf-4e1e-9a5e-fdd25209d1a5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:46.123+00:00"}

,"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.123+00:00"}

,"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.124+00:00"}

,"s":"I", "c":"-", "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":"AuthenticationFailed: Authentication failed.","replicaSet":"common-mongodb-configsvr","response":"{}"}}

{"t":\{"$date":"2022-05-18T07:02:46.124+00:00"}

,"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"common-mongodb-configsvr","host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Authentication failed."}

,"action":{"dropConnections":false,"requestImmediateCheck":false,"outcome":{"host":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017","success":false,"errorMessage":"AuthenticationFailed: Authentication failed."}}}}

{"t":\{"$date":"2022-05-18T07:02:46.124+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"5d7587f0-ce54-457f-b9ee-fc4c1e3702d5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"78de2428-eb48-4a26-95da-5962fb0f1240\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 589, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366123), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:46.126+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:46.126+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:46.126+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:46.191+00:00"}

,"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"common-mongodb-configsvr","newTopologyDescription":"{ id: \"9d8298fb-21e7-4944-abaf-543767bd0765\", topologyType: \"ReplicaSetWithPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", topologyVersion:

{ processId: ObjectId('6278a86e5ab1049272213250'), counter: 6 }

, roundTripTime: 717, lastWriteDate: new Date(1652857365000), opTime: { ts: Timestamp(1652857365, 1), t: 4 }, type: \"RSPrimary\", minWireVersion: 13, maxWireVersion: 13, me: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", setName: \"common-mongodb-configsvr\", setVersion: 2, electionId: ObjectId('7fffffff0000000000000004'), primary: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", lastUpdateTime: new Date(1652857366191), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\" }, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }","previousTopologyDescription":"{ id: \"5d7587f0-ce54-457f-b9ee-fc4c1e3702d5\", topologyType: \"ReplicaSetNoPrimary\", servers: { common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017: { address: \"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"common-mongodb-configsvr\", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId('7fffffff0000000000000004'), setVersion: 2 } }"}}

{"t":\{"$date":"2022-05-18T07:02:46.191+00:00"}

,"s":"I", "c":"SHARDING", "id":471693, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Updating the shard registry with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.191+00:00"}

,"s":"I", "c":"SHARDING", "id":22846, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Updating sharding state with confirmed replica set","attr":{"connectionString":"common-mongodb-configsvr/common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:46.195+00:00"}

,"s":"I", "c":"SHARDING", "id":4620201, "ctx":"UpdateReplicaSetOnConfigServer","msg":"Error running reload of ShardRegistry for RSM update","attr":{"error":"AuthenticationFailed: could not get updated shard list from config server :: caused by :: Authentication failed."}}

{"t":\{"$date":"2022-05-18T07:02:46.758+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:47.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:47.348+00:00"}

,"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4600}}

{"t":\{"$date":"2022-05-18T07:02:48.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:48.771+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:49.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:50.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:50.775+00:00"}

,"s":"W", "c":"SHARDING", "id":23834, "ctx":"mongosMain","msg":"Error initializing sharding state, sleeping for 2 seconds and retrying","attr":{"error":

{"code":18,"codeName":"AuthenticationFailed","errmsg":"Error loading clusterID :: caused by :: Authentication failed."}

}}

{"t":\{"$date":"2022-05-18T07:02:51.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

{"t":\{"$date":"2022-05-18T07:02:51.954+00:00"}

,"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"AuthenticationFailed: Authentication failed.","nextWakeupMillis":4800}}

{"t":\{"$date":"2022-05-18T07:02:52.117+00:00"}

,"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ShardRegistry","msg":"Connecting","attr":{"hostAndPort":"common-mongodb-configsvr-0.common-mongodb-headless.benchoneiac.svc.cluster.local:27017"}}

Details

Description

Attachments

Activity

People

Dates