I am using a sharding environment(replicaset) with multiple mongos. Sometimes when I perform query or findAndModify a collection with shardkey, there's only ONE mongos worked well, while the others will complains as below. The only way is to restart all the other mongos to fix it. However it may happen again.
Query:
Aug 19 07:09:56 dal-tsa04 Fri: Aug 19 07:09:55 [conn60192] Assertion: 13507:invalid chunk config minObj:
Aug 19 07:09:56 dal-tsa04 0x5204fa: 0x618f45 0x63a673 0x66841c 0x67d187 0x580b7c 0x6a50a0 0x35baa0673d 0x35b9ed3f6d
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a): [0x5204fa]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo12ChunkManager17getShardsForQueryERSt3setINS_5ShardESt4lessIS2_ESaIS2_EERKNS_7BSONObjE+0x8c5): [0x618f45]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0x2f3): [0x63a673]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c): [0x66841c]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77): [0x67d187]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c): [0x580b7c]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(thread_proxy+0x80): [0x6a50a0]
Aug 19 07:09:56 dal-tsa04 /lib64/libpthread.so.0: [0x35baa0673d]
Aug 19 07:09:56 dal-tsa04 /lib64/libc.so.6(clone+0x6d): [0x35b9ed3f6d]
Aug 19 07:09:56 dal-tsa04 Fri: Aug 19 07:09:55 [conn60192] AssertionException in process: invalid chunk config minObj:
FindAndModify:
Aug 19 07:10:00 dal-tsa04 Fri: Aug 19 07:10:00 [conn60177] Assertion: 8070:couldn't find a chunk aftry retry which should be impossible extracted:
{ id: "david" }Aug 19 07:10:00 dal-tsa04 0x5204fa: 0x60adfd 0x65663c 0x57df0c 0x635f52 0x66841c 0x67d187 0x580b7c 0x6a50a0 0x35baa0673d 0x35b9ed3f6d
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a): [0x5204fa]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo12ChunkManager9findChunkERKNS_7BSONObjE+0x49d): [0x60adfd]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x37c): [0x65663c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c): [0x57df0c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262): [0x635f52]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c): [0x66841c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77): [0x67d187]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c): [0x580b7c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(thread_proxy+0x80): [0x6a50a0]
Aug 19 07:10:00 dal-tsa04 /lib64/libpthread.so.0: [0x35baa0673d]
Aug 19 07:10:00 dal-tsa04 /lib64/libc.so.6(clone+0x6d): [0x35b9ed3f6d]