Sun Jan 30 13:25:50 checking replica set: set1 Sun Jan 30 13:25:50 checking replica set: set2 Sun Jan 30 13:25:50 checking replica set: set3 Sun Jan 30 13:25:57 [conn188] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:00 [conn240] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:01 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 902, sId: 3 } Sun Jan 30 13:26:01 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 902, sId: 3 } Sun Jan 30 13:26:03 [conn402] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:04 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3657, sId: 19 } Sun Jan 30 13:26:04 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3657, sId: 19 } Sun Jan 30 13:26:07 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1, sId: 64 } Sun Jan 30 13:26:07 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1, sId: 64 } Sun Jan 30 13:26:10 [conn226] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:10 checking replica set: set1 Sun Jan 30 13:26:10 checking replica set: set2 Sun Jan 30 13:26:10 checking replica set: set3 Sun Jan 30 13:26:14 [conn320] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:14 [conn102] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:15 [conn443] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:16 [conn83] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:17 [conn176] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:30 checking replica set: set1 Sun Jan 30 13:26:30 checking replica set: set2 Sun Jan 30 13:26:30 checking replica set: set3 Sun Jan 30 13:26:35 [conn359] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:41 [conn401] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:49 [conn247] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:26:50 checking replica set: set1 Sun Jan 30 13:26:50 checking replica set: set2 Sun Jan 30 13:26:50 checking replica set: set3 Sun Jan 30 13:26:55 [conn546] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:27:01 [conn228] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:27:05 [conn136] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:27:10 checking replica set: set1 Sun Jan 30 13:27:10 checking replica set: set2 Sun Jan 30 13:27:10 checking replica set: set3 Sun Jan 30 13:27:20 [conn400] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:27:30 checking replica set: set1 Sun Jan 30 13:27:30 checking replica set: set2 Sun Jan 30 13:27:30 checking replica set: set3 Sun Jan 30 13:27:43 [conn359] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:27:50 checking replica set: set1 Sun Jan 30 13:27:50 checking replica set: set2 Sun Jan 30 13:27:50 checking replica set: set3 Sun Jan 30 13:28:03 [conn499] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:28:10 checking replica set: set1 Sun Jan 30 13:28:10 checking replica set: set2 Sun Jan 30 13:28:10 checking replica set: set3 Sun Jan 30 13:28:22 [conn234] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:28:30 checking replica set: set1 Sun Jan 30 13:28:30 checking replica set: set2 Sun Jan 30 13:28:30 checking replica set: set3 Sun Jan 30 13:28:41 [conn48] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:28:50 checking replica set: set1 Sun Jan 30 13:28:50 checking replica set: set2 Sun Jan 30 13:28:50 checking replica set: set3 Sun Jan 30 13:28:55 [conn309] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:28:55 [conn309] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:29:10 checking replica set: set1 Sun Jan 30 13:29:10 checking replica set: set2 Sun Jan 30 13:29:11 checking replica set: set3 Sun Jan 30 13:29:31 checking replica set: set1 Sun Jan 30 13:29:31 checking replica set: set2 Sun Jan 30 13:29:31 checking replica set: set3 Sun Jan 30 13:29:51 checking replica set: set1 Sun Jan 30 13:29:51 checking replica set: set2 Sun Jan 30 13:29:51 checking replica set: set3 Sun Jan 30 13:29:59 [conn499] DBClientCursor::init call() failed Sun Jan 30 13:29:59 [conn499] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:29:59 [conn499] ~ScopedDBConnection: _conn != null Sun Jan 30 13:29:59 [conn499] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:29:59 [conn359] DBClientCursor::init call() failed Sun Jan 30 13:29:59 [conn359] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:29:59 [conn359] ~ScopedDBConnection: _conn != null Sun Jan 30 13:29:59 [conn359] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:29:59 [conn506] DBClientCursor::init call() failed Sun Jan 30 13:29:59 [conn506] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:29:59 [conn506] ~ScopedDBConnection: _conn != null Sun Jan 30 13:29:59 [conn506] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:29:59 [conn226] DBClientCursor::init call() failed Sun Jan 30 13:29:59 [conn226] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:29:59 [conn226] ~ScopedDBConnection: _conn != null Sun Jan 30 13:29:59 [conn226] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:29:59 [conn536] DBClientCursor::init call() failed Sun Jan 30 13:29:59 [conn536] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:29:59 [conn536] ~ScopedDBConnection: _conn != null Sun Jan 30 13:29:59 [conn536] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:00 [conn605] DBClientCursor::init call() failed Sun Jan 30 13:30:00 [conn605] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:00 [conn605] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:00 [conn605] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:00 [conn622] trying reconnect to rs3b:27018 Sun Jan 30 13:30:00 [conn622] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:30:00 [conn511] DBClientCursor::init call() failed Sun Jan 30 13:30:00 [conn511] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:00 [conn511] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:00 [conn511] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:00 [conn570] DBClientCursor::init call() failed Sun Jan 30 13:30:00 [conn570] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:00 [conn570] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:00 [conn570] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:01 [conn499] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 1018 } Sun Jan 30 13:30:01 [conn499] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 1018 } Sun Jan 30 13:30:02 [conn226] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3534, sId: 2 } Sun Jan 30 13:30:02 [conn226] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3534, sId: 2 } Sun Jan 30 13:30:03 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 508 } Sun Jan 30 13:30:03 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.servers query: { accId: 3930, sId: 2 } Sun Jan 30 13:30:03 [conn570] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 686, sId: 6 } Sun Jan 30 13:30:03 [conn539] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3391, sId: 4 } Sun Jan 30 13:30:03 [conn570] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 686, sId: 6 } Sun Jan 30 13:30:03 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2750, sId: 9 } Sun Jan 30 13:30:03 [conn594] update failed b/c of StaleConfigException, retrying left:4 ns: sd.servers query: { accId: 1007, sId: 3 } Sun Jan 30 13:30:03 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2750, sId: 9 } Sun Jan 30 13:30:03 [conn606] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3390, sId: 1 } Sun Jan 30 13:30:03 [conn606] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3390, sId: 1 } Sun Jan 30 13:30:03 [conn401] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3040, sId: 23 } Sun Jan 30 13:30:03 [conn401] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3040, sId: 23 } Sun Jan 30 13:30:04 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3589, sId: 1 } Sun Jan 30 13:30:04 [conn546] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1359, sId: 1 } Sun Jan 30 13:30:04 [conn499] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 735, sId: 9 } Sun Jan 30 13:30:04 [conn608] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 759 } Sun Jan 30 13:30:04 [conn546] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1359, sId: 1 } Sun Jan 30 13:30:04 [conn542] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1675, sId: 1 } Sun Jan 30 13:30:04 [conn608] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 759 } Sun Jan 30 13:30:04 [conn499] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 735, sId: 9 } Sun Jan 30 13:30:04 [conn542] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1675, sId: 1 } Sun Jan 30 13:30:04 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3589, sId: 1 } Sun Jan 30 13:30:04 [conn506] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3567, sId: 11 } Sun Jan 30 13:30:04 [conn400] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2349, sId: 1 } Sun Jan 30 13:30:04 [conn506] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3567, sId: 11 } Sun Jan 30 13:30:04 [conn624] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3590, sId: 2 } Sun Jan 30 13:30:04 [conn99] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3884, sId: 1 } Sun Jan 30 13:30:04 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3590, sId: 6 } Sun Jan 30 13:30:04 [conn624] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3590, sId: 2 } Sun Jan 30 13:30:04 [conn400] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2349, sId: 1 } Sun Jan 30 13:30:04 [conn99] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3884, sId: 1 } Sun Jan 30 13:30:04 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3590, sId: 6 } Sun Jan 30 13:30:04 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3344, sId: 2 } Sun Jan 30 13:30:04 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3344, sId: 2 } Sun Jan 30 13:30:04 [conn576] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2548, sId: 25 } Sun Jan 30 13:30:04 [conn576] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2548, sId: 25 } Sun Jan 30 13:30:04 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 435, sId: 1 } Sun Jan 30 13:30:04 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 435, sId: 1 } Sun Jan 30 13:30:05 [conn572] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3797, sId: 4 } Sun Jan 30 13:30:05 [conn194] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:05 [conn572] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3797, sId: 4 } Sun Jan 30 13:30:05 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2329, sId: 31 } Sun Jan 30 13:30:05 [conn532] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2609, sId: 29 } Sun Jan 30 13:30:05 [conn48] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3037, sId: 2 } Sun Jan 30 13:30:05 [conn554] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2837, sId: 4 } Sun Jan 30 13:30:05 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2329, sId: 31 } Sun Jan 30 13:30:05 [conn226] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2648, sId: 1 } Sun Jan 30 13:30:05 [conn532] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2609, sId: 29 } Sun Jan 30 13:30:05 [conn48] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3037, sId: 2 } Sun Jan 30 13:30:05 [conn554] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2837, sId: 4 } Sun Jan 30 13:30:05 [conn226] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2648, sId: 1 } Sun Jan 30 13:30:05 [conn580] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2134, sId: 1 } Sun Jan 30 13:30:05 [conn228] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 803 } Sun Jan 30 13:30:05 [conn188] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1391, sId: 10 } Sun Jan 30 13:30:05 [conn228] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 803 } Sun Jan 30 13:30:05 [conn580] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2134, sId: 1 } Sun Jan 30 13:30:05 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2862, sId: 11 } Sun Jan 30 13:30:05 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2862, sId: 11 } Sun Jan 30 13:30:05 [conn188] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1391, sId: 10 } Sun Jan 30 13:30:06 [conn234] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:06 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1438, sId: 4 } Sun Jan 30 13:30:06 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1438, sId: 4 } Sun Jan 30 13:30:06 [conn594] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3666, sId: 1 } Sun Jan 30 13:30:06 [conn102] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1466, sId: 142 } Sun Jan 30 13:30:06 [conn594] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3666, sId: 1 } Sun Jan 30 13:30:06 [conn102] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1466, sId: 142 } Sun Jan 30 13:30:06 [conn441] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3657, sId: 14 } Sun Jan 30 13:30:06 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3899, sId: 9 } Sun Jan 30 13:30:06 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3040, sId: 31 } Sun Jan 30 13:30:06 [conn627] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3146, sId: 3 } Sun Jan 30 13:30:06 [conn441] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3657, sId: 14 } Sun Jan 30 13:30:06 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3899, sId: 9 } Sun Jan 30 13:30:06 [conn511] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2876, sId: 5 } Sun Jan 30 13:30:06 [conn627] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3146, sId: 3 } Sun Jan 30 13:30:06 [conn511] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2876, sId: 5 } Sun Jan 30 13:30:06 [conn539] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2325, sId: 3 } Sun Jan 30 13:30:06 [conn282] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2329, sId: 51 } Sun Jan 30 13:30:06 [conn282] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2329, sId: 51 } Sun Jan 30 13:30:06 [conn542] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:06 [conn628] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 884 } Sun Jan 30 13:30:06 [conn628] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 884 } Sun Jan 30 13:30:06 [conn520] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 86, sId: 4 } Sun Jan 30 13:30:06 [conn520] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 86, sId: 4 } Sun Jan 30 13:30:07 [conn614] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3845, sId: 1 } Sun Jan 30 13:30:07 [conn614] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3845, sId: 1 } Sun Jan 30 13:30:07 [conn536] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2539, sId: 2 } Sun Jan 30 13:30:07 [conn536] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2539, sId: 2 } Sun Jan 30 13:30:07 [conn194] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2596, sId: 4 } Sun Jan 30 13:30:07 [conn194] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2596, sId: 4 } Sun Jan 30 13:30:07 [conn605] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1443, sId: 6 } Sun Jan 30 13:30:07 [conn605] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1443, sId: 6 } Sun Jan 30 13:30:08 [conn378] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3350, sId: 12 } Sun Jan 30 13:30:08 [conn550] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2253, sId: 1 } Sun Jan 30 13:30:08 [conn378] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3350, sId: 12 } Sun Jan 30 13:30:08 [conn247] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 966 } Sun Jan 30 13:30:08 [conn247] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 966 } Sun Jan 30 13:30:08 [conn630] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 689, sId: 4 } Sun Jan 30 13:30:08 [conn630] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 689, sId: 4 } Sun Jan 30 13:30:08 [conn550] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2253, sId: 1 } Sun Jan 30 13:30:09 [conn92] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2652, sId: 1 } Sun Jan 30 13:30:09 [conn92] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2652, sId: 1 } Sun Jan 30 13:30:09 [conn622] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3612, sId: 1 } Sun Jan 30 13:30:09 [conn622] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3612, sId: 1 } Sun Jan 30 13:30:10 [conn309] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3883, sId: 2 } Sun Jan 30 13:30:10 [conn309] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3883, sId: 2 } Sun Jan 30 13:30:11 checking replica set: set1 Sun Jan 30 13:30:11 checking replica set: set2 Sun Jan 30 13:30:11 checking replica set: set3 Sun Jan 30 13:30:11 [conn226] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:11 [conn606] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:11 [conn400] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:11 [conn606] trying reconnect to rs3b:27018 Sun Jan 30 13:30:11 [conn606] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:30:16 [conn359] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:17 [conn240] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:17 [conn570] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:18 [conn608] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:18 [conn92] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:18 [conn92] DBClientCursor::init call() failed Sun Jan 30 13:30:18 [conn92] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:18 [conn92] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:18 [conn92] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:19 [conn92] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2850, sId: 9 } Sun Jan 30 13:30:19 [conn92] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2850, sId: 9 } Sun Jan 30 13:30:19 [conn648] trying reconnect to rs3b:27018 Sun Jan 30 13:30:19 [conn648] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:30:25 [conn605] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:27 [conn282] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:28 [conn188] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:28 [conn554] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:29 [conn632] DBClientCursor::init call() failed Sun Jan 30 13:30:29 [conn632] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:29 [conn632] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:29 [conn632] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:29 [conn282] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:30:29 [conn282] DBClientCursor::init call() failed Sun Jan 30 13:30:29 [conn282] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:30:29 [conn282] ~ScopedDBConnection: _conn != null Sun Jan 30 13:30:29 [conn282] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:30:31 checking replica set: set1 Sun Jan 30 13:30:31 checking replica set: set2 Sun Jan 30 13:30:31 checking replica set: set3 Sun Jan 30 13:30:51 checking replica set: set1 Sun Jan 30 13:30:51 checking replica set: set2 Sun Jan 30 13:30:51 checking replica set: set3 Sun Jan 30 13:31:01 [conn320] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:01 [conn632] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3642, sId: 25 } Sun Jan 30 13:31:01 [conn282] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3428, sId: 1 } Sun Jan 30 13:31:01 [conn632] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3642, sId: 25 } Sun Jan 30 13:31:01 [conn282] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3428, sId: 1 } Sun Jan 30 13:31:01 [conn320] DBClientCursor::init call() failed Sun Jan 30 13:31:01 [conn320] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a04c 0x635ba6 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27ParallelSortClusteredCursor5_initEv+0x1bc) [0x58a04c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaf6) [0x635ba6] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:01 [conn320] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:01 [conn320] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:02 [conn638] DBClientCursor::init call() failed Sun Jan 30 13:31:02 [conn638] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:02 [conn638] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:02 [conn638] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:03 [conn596] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:07 [conn642] DBClientCursor::init call() failed Sun Jan 30 13:31:07 [conn642] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:07 [conn642] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:07 [conn642] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:07 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2837, sId: 4 } Sun Jan 30 13:31:07 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2837, sId: 4 } Sun Jan 30 13:31:08 [conn228] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:08 [conn228] DBClientCursor::init call() failed Sun Jan 30 13:31:08 [conn228] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:08 [conn228] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:08 [conn228] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:08 [conn48] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:09 [conn642] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2329, sId: 58 } Sun Jan 30 13:31:09 [conn642] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2329, sId: 58 } Sun Jan 30 13:31:10 [conn542] DBClientCursor::init call() failed Sun Jan 30 13:31:10 [conn542] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a04c 0x635ba6 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27ParallelSortClusteredCursor5_initEv+0x1bc) [0x58a04c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaf6) [0x635ba6] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:10 [conn542] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:10 [conn542] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:11 checking replica set: set1 Sun Jan 30 13:31:11 checking replica set: set2 Sun Jan 30 13:31:11 checking replica set: set3 Sun Jan 30 13:31:12 [conn638] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2335, sId: 6 } Sun Jan 30 13:31:12 [conn228] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3638, sId: 7 } Sun Jan 30 13:31:12 [conn542] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2127, sId: 3 } Sun Jan 30 13:31:12 [conn228] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3638, sId: 7 } Sun Jan 30 13:31:12 [conn542] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2127, sId: 3 } Sun Jan 30 13:31:13 [conn638] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3441, sId: 10 } Sun Jan 30 13:31:15 [conn550] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:15 [conn550] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:15 [conn550] DBClientCursor::init call() failed Sun Jan 30 13:31:15 [conn550] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:15 [conn550] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:15 [conn550] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:17 [conn550] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2335, sId: 9 } Sun Jan 30 13:31:17 [conn550] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2335, sId: 9 } Sun Jan 30 13:31:23 [conn282] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:23 [conn401] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:25 [conn320] DBClientCursor::init call() failed Sun Jan 30 13:31:25 [conn320] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a04c 0x635ba6 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27ParallelSortClusteredCursor5_initEv+0x1bc) [0x58a04c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaf6) [0x635ba6] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:25 [conn320] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:25 [conn320] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:28 [conn578] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:29 [conn536] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:29 [conn536] DBClientCursor::init call() failed Sun Jan 30 13:31:29 [conn536] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:29 [conn536] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:29 [conn536] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:29 [conn596] DBClientCursor::init call() failed Sun Jan 30 13:31:29 [conn596] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:29 [conn596] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:29 [conn596] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:29 [conn240] DBClientCursor::init call() failed Sun Jan 30 13:31:29 [conn240] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:29 [conn240] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:29 [conn240] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:30 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2959, sId: 3 } Sun Jan 30 13:31:30 [conn320] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2959, sId: 3 } Sun Jan 30 13:31:30 [conn309] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:30 [conn309] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:30 [conn309] DBClientCursor::init call() failed Sun Jan 30 13:31:30 [conn309] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:30 [conn309] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:30 [conn309] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:31 checking replica set: set1 Sun Jan 30 13:31:31 checking replica set: set2 Sun Jan 30 13:31:31 checking replica set: set3 Sun Jan 30 13:31:31 [conn536] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3472, sId: 1 } Sun Jan 30 13:31:31 [conn536] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3472, sId: 1 } Sun Jan 30 13:31:31 [conn176] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:31 [conn359] DBClientCursor::init call() failed Sun Jan 30 13:31:31 [conn359] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:31 [conn359] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:31 [conn359] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:31 [conn554] DBClientCursor::init call() failed Sun Jan 30 13:31:31 [conn554] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:31 [conn554] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:31 [conn554] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:31 [conn176] DBClientCursor::init call() failed Sun Jan 30 13:31:31 [conn176] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:31 [conn176] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:31 [conn176] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:31 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 1029 } Sun Jan 30 13:31:31 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 1029 } Sun Jan 30 13:31:31 [conn624] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:31 [conn624] DBClientCursor::init call() failed Sun Jan 30 13:31:31 [conn624] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:31 [conn624] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:31 [conn624] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:31 [conn646] DBClientCursor::init call() failed Sun Jan 30 13:31:31 [conn646] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:31 [conn646] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:31 [conn646] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:34 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2368, sId: 3 } Sun Jan 30 13:31:34 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2368, sId: 3 } Sun Jan 30 13:31:34 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2529, sId: 17 } Sun Jan 30 13:31:34 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2529, sId: 17 } Sun Jan 30 13:31:36 [conn646] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3170, sId: 1 } Sun Jan 30 13:31:36 [conn646] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3170, sId: 1 } Sun Jan 30 13:31:38 [conn136] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:41 [conn309] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3040, sId: 33 } Sun Jan 30 13:31:41 [conn309] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3040, sId: 33 } Sun Jan 30 13:31:45 [conn554] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1, sId: 88 } Sun Jan 30 13:31:45 [conn554] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1, sId: 88 } Sun Jan 30 13:31:46 [conn646] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:46 [conn646] DBClientCursor::init call() failed Sun Jan 30 13:31:46 [conn646] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:46 [conn646] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:46 [conn646] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:49 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1, sId: 74 } Sun Jan 30 13:31:49 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1, sId: 74 } Sun Jan 30 13:31:49 [conn624] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3912, sId: 1 } Sun Jan 30 13:31:49 [conn624] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3912, sId: 1 } Sun Jan 30 13:31:49 [conn646] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2582, sId: 3 } Sun Jan 30 13:31:49 [conn646] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2582, sId: 3 } Sun Jan 30 13:31:51 checking replica set: set1 Sun Jan 30 13:31:51 checking replica set: set2 Sun Jan 30 13:31:51 checking replica set: set3 Sun Jan 30 13:31:53 [conn532] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:56 [conn92] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:31:56 [conn681] DBClientCursor::init call() failed Sun Jan 30 13:31:56 [conn681] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:56 [conn681] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:56 [conn681] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:56 [conn234] DBClientCursor::init call() failed Sun Jan 30 13:31:56 [conn234] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:31:56 [conn234] ~ScopedDBConnection: _conn != null Sun Jan 30 13:31:56 [conn234] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:31:57 [conn606] trying reconnect to rs3b:27018 Sun Jan 30 13:32:01 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 682 } Sun Jan 30 13:32:01 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 682 } Sun Jan 30 13:32:03 [conn681] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2989, sId: 4 } Sun Jan 30 13:32:03 [conn681] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2989, sId: 4 } Sun Jan 30 13:32:11 checking replica set: set1 Sun Jan 30 13:32:11 checking replica set: set2 Sun Jan 30 13:32:11 checking replica set: set3 Sun Jan 30 13:32:17 [conn240] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:32:17 [conn240] DBClientCursor::init call() failed Sun Jan 30 13:32:17 [conn240] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:17 [conn240] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:17 [conn240] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:17 [conn677] DBClientCursor::init call() failed Sun Jan 30 13:32:17 [conn677] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:17 [conn677] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:17 [conn677] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:18 [conn596] DBClientCursor::init call() failed Sun Jan 30 13:32:18 [conn596] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:18 [conn596] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:18 [conn596] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:21 [conn234] DBClientCursor::init call() failed Sun Jan 30 13:32:21 [conn234] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a04c 0x635ba6 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27ParallelSortClusteredCursor5_initEv+0x1bc) [0x58a04c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaf6) [0x635ba6] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:21 [conn234] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:21 [conn234] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:21 [conn677] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 128, sId: 3 } Sun Jan 30 13:32:21 [conn677] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 128, sId: 3 } Sun Jan 30 13:32:21 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3249, sId: 2 } Sun Jan 30 13:32:21 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3249, sId: 2 } Sun Jan 30 13:32:23 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1437, sId: 1 } Sun Jan 30 13:32:23 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1437, sId: 1 } Sun Jan 30 13:32:23 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1820, sId: 18 } Sun Jan 30 13:32:23 [conn234] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1820, sId: 18 } Sun Jan 30 13:32:31 checking replica set: set1 Sun Jan 30 13:32:31 checking replica set: set2 Sun Jan 30 13:32:31 checking replica set: set3 Sun Jan 30 13:32:36 [conn596] DBClientCursor::init call() failed Sun Jan 30 13:32:36 [conn596] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:36 [conn596] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:36 [conn596] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:36 [conn48] DBClientCursor::init call() failed Sun Jan 30 13:32:36 [conn48] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:36 [conn48] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:36 [conn48] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:37 [conn681] DBClientCursor::init call() failed Sun Jan 30 13:32:37 [conn681] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:37 [conn681] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:37 [conn681] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:41 [conn570] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:32:41 [conn570] DBClientCursor::init call() failed Sun Jan 30 13:32:41 [conn570] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x6624f2 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x652) [0x6624f2] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:41 [conn570] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:41 [conn570] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:41 [conn531] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:32:41 [conn531] DBClientCursor::init call() failed Sun Jan 30 13:32:41 [conn531] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:41 [conn531] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:41 [conn531] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:42 [conn136] DBClientCursor::init call() failed Sun Jan 30 13:32:42 [conn136] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:42 [conn136] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:42 [conn136] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:44 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1918, sId: 1 } Sun Jan 30 13:32:44 [conn596] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1918, sId: 1 } Sun Jan 30 13:32:48 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3406, sId: 1 } Sun Jan 30 13:32:48 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3406, sId: 1 } Sun Jan 30 13:32:48 [conn402] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:32:48 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2250, sId: 6 } Sun Jan 30 13:32:48 [conn136] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2250, sId: 6 } Sun Jan 30 13:32:49 [conn578] DBClientCursor::init call() failed Sun Jan 30 13:32:49 [conn578] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:49 [conn578] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:49 [conn578] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:50 [conn630] DBClientCursor::init call() failed Sun Jan 30 13:32:50 [conn630] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:50 [conn630] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:50 [conn630] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:50 [conn570] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3567, sId: 7 } Sun Jan 30 13:32:50 [conn570] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3567, sId: 7 } Sun Jan 30 13:32:51 checking replica set: set1 Sun Jan 30 13:32:51 checking replica set: set2 Sun Jan 30 13:32:51 checking replica set: set3 Sun Jan 30 13:32:51 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 959 } Sun Jan 30 13:32:51 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 959 } Sun Jan 30 13:32:51 [conn630] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 909, sId: 824 } Sun Jan 30 13:32:51 [conn630] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 909, sId: 824 } Sun Jan 30 13:32:51 [conn681] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1644, sId: 4 } Sun Jan 30 13:32:51 [conn681] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1644, sId: 4 } Sun Jan 30 13:32:51 [conn48] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2852, sId: 1 } Sun Jan 30 13:32:51 [conn48] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2852, sId: 1 } Sun Jan 30 13:32:54 [conn630] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:32:56 [conn176] DBClientCursor::init call() failed Sun Jan 30 13:32:56 [conn176] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:56 [conn176] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:56 [conn176] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:57 [conn578] DBClientCursor::init call() failed Sun Jan 30 13:32:57 [conn578] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:32:57 [conn578] ~ScopedDBConnection: _conn != null Sun Jan 30 13:32:57 [conn578] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:32:58 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3991, sId: 1 } Sun Jan 30 13:32:58 [conn176] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3991, sId: 1 } Sun Jan 30 13:33:00 [conn684] DBClientCursor::init call() failed Sun Jan 30 13:33:00 [conn684] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:00 [conn684] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:00 [conn684] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:04 [conn684] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2480, sId: 1 } Sun Jan 30 13:33:04 [conn684] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2480, sId: 1 } Sun Jan 30 13:33:04 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2834, sId: 1 } Sun Jan 30 13:33:04 [conn578] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2834, sId: 1 } Sun Jan 30 13:33:07 [conn102] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:33:10 [conn648] trying reconnect to rs3b:27018 Sun Jan 30 13:33:11 checking replica set: set1 Sun Jan 30 13:33:11 checking replica set: set2 Sun Jan 30 13:33:11 checking replica set: set3 Sun Jan 30 13:33:23 [conn622] ns: sd.sessions ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:33:31 checking replica set: set1 Sun Jan 30 13:33:31 checking replica set: set2 Sun Jan 30 13:33:31 checking replica set: set3 Sun Jan 30 13:33:31 [conn359] DBClientCursor::init call() failed Sun Jan 30 13:33:31 [conn359] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:31 [conn359] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:31 [conn359] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:35 [conn102] DBClientCursor::init call() failed Sun Jan 30 13:33:35 [conn102] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:35 [conn102] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:35 [conn102] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:35 [conn531] DBClientCursor::init call() failed Sun Jan 30 13:33:35 [conn531] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:35 [conn531] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:35 [conn531] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:35 [conn401] DBClientCursor::init call() failed Sun Jan 30 13:33:35 [conn401] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:35 [conn401] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:35 [conn401] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:35 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3610, sId: 3 } Sun Jan 30 13:33:35 [conn359] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3610, sId: 3 } Sun Jan 30 13:33:35 [conn700] trying reconnect to rs3b:27018 Sun Jan 30 13:33:35 [conn700] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:37 [conn710] trying reconnect to rs3b:27018 Sun Jan 30 13:33:37 [conn710] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:37 [conn688] trying reconnect to rs3b:27018 Sun Jan 30 13:33:37 [conn688] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:37 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 3657, sId: 16 } Sun Jan 30 13:33:37 [conn531] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 3657, sId: 16 } Sun Jan 30 13:33:38 [conn102] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2941, sId: 1 } Sun Jan 30 13:33:38 [conn718] trying reconnect to rs3b:27018 Sun Jan 30 13:33:38 [conn718] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:38 [conn102] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2941, sId: 1 } Sun Jan 30 13:33:39 [conn712] trying reconnect to rs3b:27018 Sun Jan 30 13:33:39 [conn712] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:39 [conn359] trying reconnect to rs3b:27018 Sun Jan 30 13:33:39 [conn359] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:39 [conn630] trying reconnect to rs3b:27018 Sun Jan 30 13:33:39 [conn630] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:39 [conn240] DBClientCursor::init call() failed Sun Jan 30 13:33:39 [conn240] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:39 [conn240] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:39 [conn240] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:39 [conn704] trying reconnect to rs3b:27018 Sun Jan 30 13:33:39 [conn704] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:39 [conn642] DBClientCursor::init call() failed Sun Jan 30 13:33:39 [conn642] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:39 [conn642] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:39 [conn642] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:39 [conn402] DBClientCursor::init call() failed Sun Jan 30 13:33:39 [conn402] Assertion: 13633:error querying server: set3/rs3a:27018,rs3b:27018 0x521149 0x589818 0x58a324 0x66bb6a 0x635ef5 0x66213c 0x673674 0x57f090 0x69b480 0x302860673d 0x3027ed3f6d /home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x521149] /home/david/mongodb/latest/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0xcb8) [0x589818] /home/david/mongodb/latest/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134) [0x58a324] /home/david/mongodb/latest/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a) [0x66bb6a] /home/david/mongodb/latest/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe45) [0x635ef5] /home/david/mongodb/latest/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66213c] /home/david/mongodb/latest/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x144) [0x673674] /home/david/mongodb/latest/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x260) [0x57f090] /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69b480] /lib64/libpthread.so.0 [0x302860673d] /lib64/libc.so.6(clone+0x6d) [0x3027ed3f6d] Sun Jan 30 13:33:39 [conn402] ~ScopedDBConnection: _conn != null Sun Jan 30 13:33:39 [conn402] AssertionException in process: error querying server: set3/rs3a:27018,rs3b:27018 Sun Jan 30 13:33:40 [conn646] trying reconnect to rs3b:27018 Sun Jan 30 13:33:40 [conn646] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:40 [conn706] trying reconnect to rs3b:27018 Sun Jan 30 13:33:40 [conn706] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:40 [conn708] trying reconnect to rs3b:27018 Sun Jan 30 13:33:40 [conn708] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:40 [conn690] trying reconnect to rs3b:27018 Sun Jan 30 13:33:40 [conn690] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:40 [conn682] trying reconnect to rs3b:27018 Sun Jan 30 13:33:40 [conn682] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:41 [conn681] trying reconnect to rs3b:27018 Sun Jan 30 13:33:41 [conn681] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:41 [conn638] trying reconnect to rs3b:27018 Sun Jan 30 13:33:41 [conn638] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:41 [conn554] trying reconnect to rs3b:27018 Sun Jan 30 13:33:41 [conn554] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:41 [conn679] trying reconnect to rs3b:27018 Sun Jan 30 13:33:41 [conn679] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:42 [conn570] trying reconnect to rs3b:27018 Sun Jan 30 13:33:42 [conn570] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:42 [conn652] trying reconnect to rs3b:27018 Sun Jan 30 13:33:42 [conn652] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:46 [conn401] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2627, sId: 3 } Sun Jan 30 13:33:46 [conn401] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2627, sId: 3 } Sun Jan 30 13:33:47 [conn693] trying reconnect to rs3b:27018 Sun Jan 30 13:33:47 [conn693] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:49 [conn642] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 1540, sId: 1 } Sun Jan 30 13:33:49 [conn642] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 1540, sId: 1 } Sun Jan 30 13:33:49 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2574, sId: 1 } Sun Jan 30 13:33:49 [conn402] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2574, sId: 1 } Sun Jan 30 13:33:50 [conn691] trying reconnect to rs3b:27018 Sun Jan 30 13:33:50 [conn691] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:50 [conn698] trying reconnect to rs3b:27018 Sun Jan 30 13:33:50 [conn698] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:51 checking replica set: set1 Sun Jan 30 13:33:51 checking replica set: set2 Sun Jan 30 13:33:51 checking replica set: set3 Sun Jan 30 13:33:51 [conn683] trying reconnect to rs3b:27018 Sun Jan 30 13:33:51 [conn683] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:52 [conn694] trying reconnect to rs3b:27018 Sun Jan 30 13:33:52 [conn694] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:52 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.processesRunning query: { accId: 2883, sId: 4 } Sun Jan 30 13:33:52 [conn240] update failed b/c of StaleConfigException, retrying left:4 ns: sd.metricsLatest query: { accId: 2883, sId: 4 } Sun Jan 30 13:33:52 [conn681] trying reconnect to rs3b:27018 Sun Jan 30 13:33:52 [conn681] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:55 [conn648] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:33:55 [conn648] trying reconnect to rs3b:27018 Sun Jan 30 13:33:55 [conn648] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:34:08 [conn687] trying reconnect to rs3b:27018 Sun Jan 30 13:34:08 [conn687] reconnect rs3b:27018 ok Sun Jan 30 13:34:09 [conn624] trying reconnect to rs3b:27018 Sun Jan 30 13:34:09 [conn624] reconnect rs3b:27018 ok Sun Jan 30 13:34:09 [conn686] trying reconnect to rs3b:27018 Sun Jan 30 13:34:09 [conn686] reconnect rs3b:27018 ok Sun Jan 30 13:34:11 [conn532] trying reconnect to rs3b:27018 Sun Jan 30 13:34:11 [conn532] reconnect rs3b:27018 ok Sun Jan 30 13:34:11 checking replica set: set1 Sun Jan 30 13:34:11 checking replica set: set2 Sun Jan 30 13:34:11 checking replica set: set3 Sun Jan 30 13:34:13 [conn309] trying reconnect to rs3b:27018 Sun Jan 30 13:34:13 [conn320] trying reconnect to rs3b:27018 Sun Jan 30 13:34:13 [conn309] reconnect rs3b:27018 ok Sun Jan 30 13:34:13 [conn320] reconnect rs3b:27018 ok Sun Jan 30 13:34:13 [conn680] trying reconnect to rs3b:27018 Sun Jan 30 13:34:13 [conn680] reconnect rs3b:27018 ok Sun Jan 30 13:34:13 [conn689] trying reconnect to rs3b:27018 Sun Jan 30 13:34:13 [conn689] reconnect rs3b:27018 ok Sun Jan 30 13:34:15 [conn646] trying reconnect to rs3b:27018 Sun Jan 30 13:34:15 [conn646] reconnect rs3b:27018 ok Sun Jan 30 13:34:15 [conn679] trying reconnect to rs3b:27018 Sun Jan 30 13:34:15 [conn679] reconnect rs3b:27018 ok Sun Jan 30 13:34:15 [conn550] trying reconnect to rs3b:27018 Sun Jan 30 13:34:15 [conn550] reconnect rs3b:27018 ok Sun Jan 30 13:34:15 [conn632] trying reconnect to rs3b:27018 Sun Jan 30 13:34:15 [conn632] reconnect rs3b:27018 ok Sun Jan 30 13:34:16 [conn712] trying reconnect to rs3b:27018 Sun Jan 30 13:34:16 [conn712] reconnect rs3b:27018 ok Sun Jan 30 13:34:16 [conn638] trying reconnect to rs3b:27018 Sun Jan 30 13:34:16 [conn698] trying reconnect to rs3b:27018 Sun Jan 30 13:34:16 [conn638] reconnect rs3b:27018 ok Sun Jan 30 13:34:16 [conn698] reconnect rs3b:27018 ok Sun Jan 30 13:34:16 [conn658] trying reconnect to rs3b:27018 Sun Jan 30 13:34:16 [conn658] reconnect rs3b:27018 ok Sun Jan 30 13:34:16 [conn630] trying reconnect to rs3b:27018 Sun Jan 30 13:34:16 [conn630] reconnect rs3b:27018 ok Sun Jan 30 13:34:22 [conn685] trying reconnect to rs3b:27018 Sun Jan 30 13:34:22 [conn685] reconnect rs3b:27018 ok Sun Jan 30 13:34:22 [conn656] trying reconnect to rs3b:27018 Sun Jan 30 13:34:22 [conn656] reconnect rs3b:27018 ok Sun Jan 30 13:34:23 [conn644] trying reconnect to rs3b:27018 Sun Jan 30 13:34:23 [conn644] reconnect rs3b:27018 ok Sun Jan 30 13:34:23 [conn682] trying reconnect to rs3b:27018 Sun Jan 30 13:34:23 [conn682] reconnect rs3b:27018 ok Sun Jan 30 13:34:31 checking replica set: set1 Sun Jan 30 13:34:31 checking replica set: set2 Sun Jan 30 13:34:31 checking replica set: set3 Sun Jan 30 13:34:32 [conn282] trying reconnect to rs3b:27018 Sun Jan 30 13:34:32 [conn282] reconnect rs3b:27018 ok Sun Jan 30 13:34:32 [conn710] trying reconnect to rs3b:27018 Sun Jan 30 13:34:32 [conn710] reconnect rs3b:27018 ok Sun Jan 30 13:34:39 [conn542] trying reconnect to rs3b:27018 Sun Jan 30 13:34:39 [conn542] reconnect rs3b:27018 ok Sun Jan 30 13:34:40 [conn654] trying reconnect to rs3b:27018 Sun Jan 30 13:34:40 [conn654] reconnect rs3b:27018 ok Sun Jan 30 13:34:41 [conn704] trying reconnect to rs3b:27018 Sun Jan 30 13:34:41 [conn704] reconnect rs3b:27018 ok Sun Jan 30 13:34:42 [conn696] trying reconnect to rs3b:27018 Sun Jan 30 13:34:42 [conn696] reconnect rs3b:27018 ok Sun Jan 30 13:34:51 checking replica set: set1 Sun Jan 30 13:34:51 checking replica set: set2 Sun Jan 30 13:34:51 checking replica set: set3 Sun Jan 30 13:35:05 [conn92] trying reconnect to rs3b:27018 Sun Jan 30 13:35:05 [conn92] reconnect rs3b:27018 ok Sun Jan 30 13:35:06 [conn606] reconnect rs3b:27018 failed couldn't connect to server rs3b:27018 Sun Jan 30 13:35:06 [conn606] trying reconnect to rs3b:27018 Sun Jan 30 13:35:06 [conn606] reconnect rs3b:27018 ok Sun Jan 30 13:35:06 [conn660] trying reconnect to rs3b:27018 Sun Jan 30 13:35:06 [conn660] reconnect rs3b:27018 ok Sun Jan 30 13:35:11 checking replica set: set1 Sun Jan 30 13:35:11 checking replica set: set2 Sun Jan 30 13:35:11 checking replica set: set3 Sun Jan 30 13:35:11 [conn531] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:35:31 [conn401] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:35:31 checking replica set: set1 Sun Jan 30 13:35:31 checking replica set: set2 Sun Jan 30 13:35:31 checking replica set: set3 Sun Jan 30 13:35:36 [conn691] trying reconnect to rs3b:27018 Sun Jan 30 13:35:36 [conn691] reconnect rs3b:27018 ok Sun Jan 30 13:35:37 [conn718] trying reconnect to rs3b:27018 Sun Jan 30 13:35:37 [conn718] reconnect rs3b:27018 ok Sun Jan 30 13:35:51 checking replica set: set1 Sun Jan 30 13:35:51 checking replica set: set2 Sun Jan 30 13:35:51 checking replica set: set3 Sun Jan 30 13:35:55 [conn48] MessagingPort say send() errno:32 Broken pipe 10.121.14.8:27018 Sun Jan 30 13:35:55 [conn48] trying reconnect to rs3b:27018 Sun Jan 30 13:35:55 [conn48] reconnect rs3b:27018 ok Sun Jan 30 13:36:11 checking replica set: set1 Sun Jan 30 13:36:11 checking replica set: set2 Sun Jan 30 13:36:11 checking replica set: set3 Sun Jan 30 13:36:20 [conn693] trying reconnect to rs3b:27018 Sun Jan 30 13:36:20 [conn693] reconnect rs3b:27018 ok Sun Jan 30 13:36:26 [conn240] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:36:31 checking replica set: set1 Sun Jan 30 13:36:31 checking replica set: set2 Sun Jan 30 13:36:31 checking replica set: set3 Sun Jan 30 13:36:32 [conn708] trying reconnect to rs3b:27018 Sun Jan 30 13:36:32 [conn708] reconnect rs3b:27018 ok Sun Jan 30 13:36:36 [conn726] trying reconnect to rs3b:27018 Sun Jan 30 13:36:36 [conn726] reconnect rs3b:27018 ok Sun Jan 30 13:36:37 [conn648] trying reconnect to rs3b:27018 Sun Jan 30 13:36:37 [conn648] reconnect rs3b:27018 ok Sun Jan 30 13:36:38 [conn570] trying reconnect to rs3b:27018 Sun Jan 30 13:36:38 [conn570] reconnect rs3b:27018 ok Sun Jan 30 13:36:38 [conn722] trying reconnect to rs3b:27018 Sun Jan 30 13:36:38 [conn722] reconnect rs3b:27018 ok Sun Jan 30 13:36:47 [conn714] MessagingPort say send() errno:32 Broken pipe 10.121.14.8:27018 Sun Jan 30 13:36:47 [conn714] trying reconnect to rs3b:27018 Sun Jan 30 13:36:47 [conn714] reconnect rs3b:27018 ok Sun Jan 30 13:36:51 checking replica set: set1 Sun Jan 30 13:36:51 checking replica set: set2 Sun Jan 30 13:36:51 checking replica set: set3 Sun Jan 30 13:37:10 [conn683] trying reconnect to rs3b:27018 Sun Jan 30 13:37:10 [conn683] reconnect rs3b:27018 ok Sun Jan 30 13:37:11 checking replica set: set1 Sun Jan 30 13:37:11 checking replica set: set2 Sun Jan 30 13:37:11 checking replica set: set3 Sun Jan 30 13:37:31 checking replica set: set1 Sun Jan 30 13:37:31 checking replica set: set2 Sun Jan 30 13:37:31 checking replica set: set3 Sun Jan 30 13:37:39 [conn622] trying reconnect to rs3b:27018 Sun Jan 30 13:37:39 [conn622] reconnect rs3b:27018 ok Sun Jan 30 13:37:51 checking replica set: set1 Sun Jan 30 13:37:51 checking replica set: set2 Sun Jan 30 13:37:51 checking replica set: set3 Sun Jan 30 13:38:11 checking replica set: set1 Sun Jan 30 13:38:11 checking replica set: set2 Sun Jan 30 13:38:11 checking replica set: set3 Sun Jan 30 13:38:31 checking replica set: set1 Sun Jan 30 13:38:31 checking replica set: set2 Sun Jan 30 13:38:31 checking replica set: set3 Sun Jan 30 13:38:40 [conn677] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0 Sun Jan 30 13:38:51 checking replica set: set1 Sun Jan 30 13:38:51 checking replica set: set2 Sun Jan 30 13:38:51 checking replica set: set3