[SERVER-3652] Sometimes got "invalid chunk config minObj" or "Assertion: 8070:couldn't find a chunk aftry retry which should be impossible extracted" when there're multiple mongos Created: 19/Aug/11  Updated: 11/Dec/18  Resolved: 02/Nov/11

Status: Closed
Project: Core Server
Component/s: Querying, Sharding
Affects Version/s: 1.8.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Edward Wei Assignee: Scott Hernandez (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: mongos
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 5 64bit


Operating System: Linux
Participants:

 Description   

I am using a sharding environment(replicaset) with multiple mongos. Sometimes when I perform query or findAndModify a collection with shardkey, there's only ONE mongos worked well, while the others will complains as below. The only way is to restart all the other mongos to fix it. However it may happen again.

Query:
Aug 19 07:09:56 dal-tsa04 Fri: Aug 19 07:09:55 [conn60192] Assertion: 13507:invalid chunk config minObj:

{ id: "newbie" }

Aug 19 07:09:56 dal-tsa04 0x5204fa: 0x618f45 0x63a673 0x66841c 0x67d187 0x580b7c 0x6a50a0 0x35baa0673d 0x35b9ed3f6d
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a): [0x5204fa]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo12ChunkManager17getShardsForQueryERSt3setINS_5ShardESt4lessIS2_ESaIS2_EERKNS_7BSONObjE+0x8c5): [0x618f45]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0x2f3): [0x63a673]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c): [0x66841c]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77): [0x67d187]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c): [0x580b7c]
Aug 19 07:09:56 dal-tsa04 /usr/bin/mongos(thread_proxy+0x80): [0x6a50a0]
Aug 19 07:09:56 dal-tsa04 /lib64/libpthread.so.0: [0x35baa0673d]
Aug 19 07:09:56 dal-tsa04 /lib64/libc.so.6(clone+0x6d): [0x35b9ed3f6d]
Aug 19 07:09:56 dal-tsa04 Fri: Aug 19 07:09:55 [conn60192] AssertionException in process: invalid chunk config minObj:

{ id: "newbie" }

FindAndModify:

Aug 19 07:10:00 dal-tsa04 Fri: Aug 19 07:10:00 [conn60177] Assertion: 8070:couldn't find a chunk aftry retry which should be impossible extracted:

{ id: "david" }

Aug 19 07:10:00 dal-tsa04 0x5204fa: 0x60adfd 0x65663c 0x57df0c 0x635f52 0x66841c 0x67d187 0x580b7c 0x6a50a0 0x35baa0673d 0x35b9ed3f6d
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x12a): [0x5204fa]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo12ChunkManager9findChunkERKNS_7BSONObjE+0x49d): [0x60adfd]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x37c): [0x65663c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c): [0x57df0c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262): [0x635f52]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c): [0x66841c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77): [0x67d187]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c): [0x580b7c]
Aug 19 07:10:00 dal-tsa04 /usr/bin/mongos(thread_proxy+0x80): [0x6a50a0]
Aug 19 07:10:00 dal-tsa04 /lib64/libpthread.so.0: [0x35baa0673d]
Aug 19 07:10:00 dal-tsa04 /lib64/libc.so.6(clone+0x6d): [0x35b9ed3f6d]



 Comments   
Comment by Edward Wei [ 01/Nov/11 ]

@Ian, I've not seen these kind of errors again, maybe I had connection issue at that time. I would suggest close this case first, thank you for help.

Comment by Ian Whalen (Inactive) [ 31/Oct/11 ]

@edward, have you seen the same errors again? If so it would be useful if you could attach the logs.

Comment by Edward Wei [ 23/Aug/11 ]

@Scott,
I am sure that all the config server is running. I've set the mongos in verbose mode, I'll attach the logs if the same errors are reproduced.

Comment by Scott Hernandez (Inactive) [ 19/Aug/11 ]

Are all your config servers up and running? Please attach more of the mongos logs before the errors. If you can enable more verbose logging that would help as well.

Generated at Thu Feb 08 03:03:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.