On one of our test servers, I am seeing exceptions when trying to query mongo. The server was idle for over 12 hours before I attempted these queries.

The PHP driver is reporting these exceptions:

1) DBClientBase::findOne: transport error: rs1a:27018 query:

{ setShardVersion: "sd.sessions", configdb: "config1:27019", version: Timestamp 4000|1, serverID: ObjectId('4d4d074513e250e07d6e3a4c'), shard: "shard1", shardHost: "set1/rs1a:27018,rs1b:27018" }

2) error querying server: set1/rs1a:27018,rs1b:27018

Set 1 has changed since the last access with a new member being added for debugging CS-303.

mongos log shows:

Tue Feb 8 12:14:50 checking replica set: set3
Tue Feb 8 12:15:10 checking replica set: set1
Tue Feb 8 12:15:10 checking replica set: set2
Tue Feb 8 12:15:10 checking replica set: set3
Tue Feb 8 12:15:30 checking replica set: set1
Tue Feb 8 12:15:30 checking replica set: set2
Tue Feb 8 12:15:30 checking replica set: set3
Tue Feb 8 12:15:37 [conn238] creating WriteBackListener for: set1/rs1a:27018,rs1b:27018,rs1e:27018
Tue Feb 8 12:15:37 [conn238] MessagingPort recv() errno:104 Connection reset by peer 10.121.14.3:27018
Tue Feb 8 12:15:37 [conn238] SocketException: remote: 10.121.14.3:27018 error: 9001 socket exception [1]
Tue Feb 8 12:15:37 [conn238] DBClientCursor::init call() failed
Tue Feb 8 12:15:38 [conn238] update failed b/c of StaleConfigException, retrying left:4 ns: sd.sessions query:

{ accId: 1, sessId: "45df4rdeifaqq74ukrvgrm1np5" }

Tue Feb 8 12:15:45 [conn164] MessagingPort recv() errno:104 Connection reset by peer 10.121.14.3:27018
Tue Feb 8 12:15:45 [conn164] SocketException: remote: 10.121.14.3:27018 error: 9001 socket exception [1]
Tue Feb 8 12:15:45 [conn164] DBClientCursor::init call() failed
Tue Feb 8 12:15:45 [conn164] Assertion: 13633:error querying server: set1/rs1a:27018,rs1b:27018
0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d
/home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090]
/home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480]
/lib64/libpthread.so.0 [0x302860673d]
/lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d]
Tue Feb 8 12:15:45 [conn164] AssertionException in process: error querying server: set1/rs1a:27018,rs1b:27018
Tue Feb 8 12:15:45 [conn164] update failed b/c of StaleConfigException, retrying left:4 ns: sd.sessions query:

{ accId: 1, sessId: "45df4rdeifaqq74ukrvgrm1np5" }

Tue Feb 8 12:15:48 [conn338] MessagingPort recv() errno:104 Connection reset by peer 10.121.14.3:27018
Tue Feb 8 12:15:48 [conn338] SocketException: remote: 10.121.14.3:27018 error: 9001 socket exception [1]
Tue Feb 8 12:15:48 [conn338] DBClientCursor::init call() failed
Tue Feb 8 12:15:48 [conn338] Assertion: 13633:error querying server: set1/rs1a:27018,rs1b:27018
0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d
/home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090]
/home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480]
/lib64/libpthread.so.0 [0x302860673d]
/lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d]
Tue Feb 8 12:15:48 [conn338] AssertionException in process: error querying server: set1/rs1a:27018,rs1b:27018
Tue Feb 8 12:15:48 [conn338] update failed b/c of StaleConfigException, retrying left:4 ns: sd.sessions query:

{ accId: 1, sessId: "45df4rdeifaqq74ukrvgrm1np5" }

Tue Feb 8 12:15:50 checking replica set: set1
Tue Feb 8 12:15:50 checking replica set: set2
Tue Feb 8 12:15:50 checking replica set: set3
Tue Feb 8 12:16:10 checking replica set: set1
Tue Feb 8 12:16:10 checking replica set: set2
Tue Feb 8 12:16:10 checking replica set: set3
Tue Feb 8 12:16:27 [conn302] MessagingPort recv() errno:104 Connection reset by peer 10.121.14.3:27018
Tue Feb 8 12:16:27 [conn302] SocketException: remote: 10.121.14.3:27018 error: 9001 socket exception [1]
Tue Feb 8 12:16:27 [conn302] DBClientCursor::init call() failed
Tue Feb 8 12:16:27 [conn302] update failed b/c of StaleConfigException, retrying left:4 ns: sd.sessions query:

{ accId: 1, sessId: "45df4rdeifaqq74ukrvgrm1np5" }

Tue Feb 8 12:16:30 checking replica set: set1
Tue Feb 8 12:16:30 checking replica set: set2
Tue Feb 8 12:16:30 checking replica set: set3