[SERVER-3040] Too many ReplicaSetMonitor::_checkConnection: messages in mongos.log after replica set failover Created: 04/May/11  Updated: 30/Mar/12  Resolved: 25/May/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.8.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Michael Conigliaro Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC2


Issue Links:
Depends
Duplicate
is duplicated by SERVER-3303 uncaught exception: error: { "$err" :... Closed
Operating System: Linux
Participants:

 Description   

After replica set failover, why do I see about 30 "ReplicaSetMonitor::_checkConnection:" messages per second in my mongos log? This seems to go on until I fail back to the original master. Here's an example message:

Wed May 4 23:09:52 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

 Comments   
Comment by Michael Conigliaro [ 16/May/11 ]

OK, great. That's all I needed to know.

Comment by Eliot Horowitz (Inactive) [ 16/May/11 ]

Those messages are what we are expecting.
They may be more verbose then you want, but there is nothing wrong going on.

Comment by Michael Conigliaro [ 16/May/11 ]

The latest nightly definitely seems to have reduced the number of these messages, but I'm still seeing an increased number of them long after failover has completed. I've just been running rs.stepDown(); on the masters in shard 2. I don't know if these messages mean anything, but I figured I'd mention it. And just like before, once I fail back to the original master on shard 2, all these messages go away (or happen much less frequently).

Mon May 16 18:06:04 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb10.example.com:27018

{ setName: "4", ismaster: true, secondary: false, hosts: [ "mongodb10.example.com:27018", "mongodb11.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 1
Mon May 16 18:06:04 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 2
Mon May 16 18:06:04 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 3
Mon May 16 18:06:04 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb07.example.com:27018

{ setName: "3", ismaster: true, secondary: false, hosts: [ "mongodb07.example.com:27018", "mongodb08.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 4
Mon May 16 18:06:05 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:05 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:07 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:07 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:07 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:07 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:08 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:08 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:09 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:09 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:10 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:10 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:10 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:10 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:12 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:12 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:12 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:12 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:14 [Balancer] skipping balancing round because balancing is disabled
Mon May 16 18:06:14 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:14 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:14 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:14 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:19 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:19 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:19 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:19 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:21 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:21 [conn1] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:23 [conn5] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:23 [conn5] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:24 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:24 [conn23] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

_check : 2/mongodb05.example.com:27018,mongodb04.example.com:27018
Mon May 16 18:06:24 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb10.example.com:27018

{ setName: "4", ismaster: true, secondary: false, hosts: [ "mongodb10.example.com:27018", "mongodb11.example.com:27018" ], arbiters: [ "mongodb12.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 1
Mon May 16 18:06:24 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb01.example.com:27018

{ setName: "1", ismaster: true, secondary: false, hosts: [ "mongodb01.example.com:27018", "mongodb02.example.com:27018" ], arbiters: [ "mongodb03.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 2
Mon May 16 18:06:24 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb05.example.com:27018

{ setName: "2", ismaster: true, secondary: false, hosts: [ "mongodb05.example.com:27018", "mongodb04.example.com:27018" ], arbiters: [ "mongodb06.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 3
Mon May 16 18:06:24 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: mongodb07.example.com:27018

{ setName: "3", ismaster: true, secondary: false, hosts: [ "mongodb07.example.com:27018", "mongodb08.example.com:27018" ], arbiters: [ "mongodb09.example.com:27018" ], maxBsonObjectSize: 16777216, ok: 1.0 }

checking replica set: 4

Comment by Michael Conigliaro [ 09/May/11 ]

FYI, this may be related to https://jira.mongodb.org/browse/SERVER-3049.

Comment by Michael Conigliaro [ 09/May/11 ]

> db.printShardingStatus()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:
{
"_id" : "2",
"host" : "2/mongodb05.example.com:27018,mongodb04.example.com:27018"
}
{
"_id" : "1",
"host" : "1/mongodb01.example.com:27018,mongodb02.example.com:27018"
}
{
"_id" : "3",
"host" : "3/mongodb07.example.com:27018,mongodb08.example.com:27018"
}
{
"_id" : "4",
"host" : "4/mongodb10.example.com:27018,mongodb11.example.com:27018"
}
databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "test", "partitioned" : false, "primary" : "2" } { "_id" : "example", "partitioned" : true, "primary" : "2" }

example.stats chunks:
4 22
3 24
1 22
2 23
too many chunksn to print, use verbose if you want to force print
example.userEventsJournal chunks:
4 194
1 202
3 189
2 172
too many chunksn to print, use verbose if you want to force print
example.userfile chunks:
4 2
2 5
{ "userId" :

{ $minKey : 1 }

} -->>

{ "userId" : "000009a6-02e5-4277-90a3-fb38264440f9" }

on : 4

{ "t" : 7000, "i" : 0 }

{ "userId" : "000009a6-02e5-4277-90a3-fb38264440f9" }

-->>

{ "userId" : "3295759f-14d4-46b3-bcac-22e6b8c35400" }

on : 2

{ "t" : 8000, "i" : 2 }

{ "userId" : "3295759f-14d4-46b3-bcac-22e6b8c35400" }

-->>

{ "userId" : "651a8cb3-7df8-43bf-bbee-6c58a1f88b28" }

on : 2

{ "t" : 8000, "i" : 4 }

{ "userId" : "651a8cb3-7df8-43bf-bbee-6c58a1f88b28" }

-->>

{ "userId" : "97cf4d36-c30d-4888-bd26-6a3ef8dcb662" }

on : 2

{ "t" : 8000, "i" : 6 }

{ "userId" : "97cf4d36-c30d-4888-bd26-6a3ef8dcb662" }

-->>

{ "userId" : "ca921ed8-67bb-44e7-a802-8c48a8ae8c8d" }

on : 2

{ "t" : 8000, "i" : 8 }

{ "userId" : "ca921ed8-67bb-44e7-a802-8c48a8ae8c8d" }

-->>

{ "userId" : "testuser2" }

on : 2

{ "t" : 8000, "i" : 9 }

{ "userId" : "testuser2" }

-->> { "userId" :

{ $maxKey : 1 }

} on : 4

{ "t" : 8000, "i" : 0 } { "_id" : "local", "partitioned" : false, "primary" : "2" }
Comment by Eliot Horowitz (Inactive) [ 09/May/11 ]

Can you send the output of db.printShardingStatus()

Generated at Thu Feb 08 03:01:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.