[SERVER-5150] drop collection : now errors with sharding Created: 29/Feb/12  Updated: 08/Apr/12  Resolved: 03/Apr/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.2
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Brett Goldstein Assignee: Greg Studer
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-4262 when dropping collections need to inv... Closed
Operating System: ALL
Participants:

 Description   

I added sharding to a collection with ~ 3 million documents but as the balancing seemed to have a performance impact, I dropped the collection.

I added the shard key and index and as data was fed back in, errors started to appear. This includes from the shell:

mongos> db.cta.count()
Wed Feb 29 16:58:21 uncaught exception: count failed: {
"assertion" : "setShardVersion failed host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: \"assertion s/d_state.cpp:529\", errmsg: \"db assertion failure\", ok: 0.0 }

",
"assertionCode" : 10429,
"errmsg" : "db assertion failure",
"ok" : 0
}

From mongos, I have the following:

Wed Feb 29 16:58:05 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4ead7d2d46e699744598a6
Wed Feb 29 16:58:05 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Wed Feb 29 16:58:15 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4ead872d46e699744598a7
Wed Feb 29 16:58:15 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Wed Feb 29 16:58:21 [conn1869] going to retry checkShardVersion host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Wed Feb 29 16:58:21 [conn1869] going to retry checkShardVersion host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Wed Feb 29 16:58:21 [conn1869] going to retry checkShardVersion host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Wed Feb 29 16:58:21 [conn1869] going to retry checkShardVersion host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Wed Feb 29 16:58:21 [conn1869] setShardVersion failed host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Wed Feb 29 16:58:21 [conn1869] Assertion: 10429:setShardVersion failed host: mongos3a:27018

{ oldVersion: Timestamp 9000|0, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

0x535572 0x7f4556 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x5c1e06 0x5bffe7 0x79879b 0x793175 0x76cf2b 0x7b4927 0x7c6cf1 0x5e6a07 0x3144a0673d 0x3143ed44bd
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x535572]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f4556]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16) [0x5c1e06]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0x137) [0x5bffe7]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo15dbgrid_pub_cmds8CountCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x6ab) [0x79879b]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x8b5) [0x793175]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5cb) [0x76cf2b]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b4927]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x7c6cf1]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x5e6a07]
/lib64/libpthread.so.0 [0x3144a0673d]
/lib64/libc.so.6(clone+0x6d) [0x3143ed44bd]
Wed Feb 29 16:58:21 [conn1869] ~ScopedDBConnection: _conn != null
Wed Feb 29 16:58:25 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4ead912d46e699744598a8
Wed Feb 29 16:58:25 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Wed Feb 29 16:58:35 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4ead9b2d46e699744598a9
Wed Feb 29 16:58:35 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Wed Feb 29 16:58:45 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4eada52d46e699744598aa
Wed Feb 29 16:58:46 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Wed Feb 29 16:58:56 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4eadb02d46e699744598ab
Wed Feb 29 16:58:56 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.

Any ideas?

thx



 Comments   
Comment by Daniel Pasette (Inactive) [ 03/Apr/12 ]

this duplicates SERVER-4262

Comment by Daniel Pasette (Inactive) [ 03/Apr/12 ]

Root issue is: SERVER-4262.

To workaround, try running on all mongos: db.adminCommand("flushRouterConfig").

Comment by Brett Goldstein [ 01/Mar/12 ]

this showed up in the logs this morning. No known precursing action:

Thu Mar 1 08:08:40 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4f82e82d46e6997445b7b2
Thu Mar 1 08:08:40 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Thu Mar 1 08:08:45 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4f82ed2d46e6997445b7b3
Thu Mar 1 08:08:45 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Thu Mar 1 08:08:50 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' acquired, ts : 4f4f82f22d46e6997445b7b4
Thu Mar 1 08:08:50 [Balancer] distributed lock 'balancer/my07apl01.cityofchicago.org:27017:1330355727:1804289383' unlocked.
Thu Mar 1 08:08:55 [conn411] going to retry checkShardVersion host: mongos1a:27018

{ oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Thu Mar 1 08:08:55 [conn411] going to retry checkShardVersion host: mongos1a:27018

{ oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Thu Mar 1 08:08:55 [conn411] going to retry checkShardVersion host: mongos1a:27018

{ oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Thu Mar 1 08:08:55 [conn411] going to retry checkShardVersion host: mongos1a:27018

{ oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Thu Mar 1 08:08:55 [conn411] setShardVersion failed host: mongos1a:27018

{ oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0 }

Thu Mar 1 08:08:55 [conn411] Assertion: 10429:setShardVersion failed host: mongos1a:27018 { oldVersion: Timestamp 272000|1, assertion: "assertion s/d_state.cpp:529", errmsg: "db assertion failure", ok: 0.0
}
0x535572 0x7f4556 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x7f3d2d 0x5c1e06 0x5bffe7 0x5c36f9 0x7a616d 0x793175 0x76cf2b 0x7b4927 0x7c6cf1 0x5e6a07 0x3144a0673d 0x3143ed44bd
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x535572]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f4556]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos [0x7f3d2d]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16) [0x5c1e06]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0x137) [0x5bffe7]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo15ShardConnection3getEv+0x9) [0x5c36f9]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo15dbgrid_pub_cmds5MRCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc5d) [0x7a616d]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x8b5) [0x793175]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5cb) [0x76cf2b]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b4927]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x7c6cf1]
./mongodb-linux-x86_64-2.0.2/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x5e6a07]
/lib64/libpthread.so.0 [0x3144a0673d]
/lib64/libc.so.6(clone+0x6d) [0x3143ed44bd]
Thu Mar 1 08:08:55 [conn411] ~ScopedDBConnection: _conn != null

Generated at Thu Feb 08 03:08:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.