[SERVER-2828] Sharding status not communicated across Routers (mongoS) Created: 24/Mar/11  Updated: 12/Jul/16  Resolved: 19/May/11

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 1.8.0
Fix Version/s: 1.9.1

Type: Bug Priority: Major - P3
Reporter: Gaetan Voyer-Perrault Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
Operating System: ALL
Participants:

 Description   

Repro on 1.8.0.

1. Start sharded set
2. Start config DB
3. Start 2 MongoS both pointing at config DB
4. Open 2 windows
4.a. mongoS #1
4.b. mongoS #2

In #2
> use infinity
> db.events_test.stats()

In #1
> use admin
> db.runCommand(

{enablesharding : "infinity"}

)
> db.runCommand({shardcollection: "infinity.events_test", key: {_id: 1}});
> db.events_test.stats()

In #2
> db.events_test.stats()

===
Bug here
The first thing you'll see here is that #1 & #2 do not match.
#1 will say the collection is sharded
#2 will say the collection is not sharded
===

More fun:
#1
> db.events_test.ensureIndex(

{event_name : 1}

);
> db.events_test.stats()

#2
> db.events_test.stats()

Notice how the indexes were properly communicated?

So we somehow communicated the indexes, but not the sharding status.
I can write to both collections and the data appears.

===
Resolution: bouncing mongoS solves the problem
===



 Comments   
Comment by auto [ 19/May/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: fix moveChunk on new mongos SERVER-2828
Branch: master
https://github.com/mongodb/mongo/commit/096e570a64bef4c6aeda43a400ca16eb7151ec5f

Comment by auto [ 19/May/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: throw StaleConfigException for passthrough command failure with stale config SERVER-2828
Branch: master
https://github.com/mongodb/mongo/commit/ea1a8f4c295460916adb4c4024ded37750a1267b

Comment by auto [ 19/May/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: put all connections that are assigned to ShardConnection into sharded state immediately on creation SERVER-2828
Branch: master
https://github.com/mongodb/mongo/commit/a9924101a7c22e4919d28d1eaa518d49d64e9de1

Comment by Eliot Horowitz (Inactive) [ 06/Apr/11 ]

The correct fix for this is fairly complex, and I dont want to hack it too badly.

I added a command to flush the config on a mongos manually
SERVER-2666

{ flushRouterConfig : 1 }

With this, after you call shardcollection, you can call that on each mongos, and it'll work around the issus.
Will backport SERVER-2666 for 1.8.2

Comment by David Mytton [ 06/Apr/11 ]

We have new collections created each day and they're automatically sharded. This bug just caused one of our mongos to assert on queries:

Wed Apr 6 00:00:53 [WriteBackListener] Assertion: 10181:not sharded:sd.processes_110406
0x51f4a9 0x5ecf19 0x69703a 0x5035ab 0x504e64 0x69ec30 0x341ee0673d 0x341e2d3f6d
/home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x51f4a9]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsb+0x119) [0x5ecf19]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo17WriteBackListener3runEv+0xaea) [0x69703a]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b) [0x5035ab]
/home/david/mongodb/latest/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x504e64]
/home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69ec30]
/lib64/libpthread.so.0 [0x341ee0673d]
/lib64/libc.so.6(clone+0x6d) [0x341e2d3f6d]
Wed Apr 6 00:00:53 [WriteBackListener] ~ScopedDbConnection: _conn != null
Wed Apr 6 00:00:53 [WriteBackListener] WriteBackListener exception : St9exception
Wed Apr 6 00:00:54 [WriteBackListener] Assertion: 10181:not sharded:sd.processes_110406
0x51f4a9 0x5ecf19 0x69703a 0x5035ab 0x504e64 0x69ec30 0x341ee0673d 0x341e2d3f6d
/home/david/mongodb/latest/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x51f4a9]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo8DBConfig15getChunkManagerERKSsb+0x119) [0x5ecf19]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo17WriteBackListener3runEv+0xaea) [0x69703a]
/home/david/mongodb/latest/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b) [0x5035ab]
/home/david/mongodb/latest/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x504e64]
/home/david/mongodb/latest/bin/mongos(thread_proxy+0x80) [0x69ec30]
/lib64/libpthread.so.0 [0x341ee0673d]
/lib64/libc.so.6(clone+0x6d) [0x341e2d3f6d]

Comment by David Mytton [ 05/Apr/11 ]

+1 for this in 1.8

Comment by Mathias Stearn [ 05/Apr/11 ]

There is a similar issue with settings such as chunksize. There is currently no way to reload them without bouncing a mongos.

Comment by auto [ 31/Mar/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: separate test for SERVER-2828
Branch: master
https://github.com/mongodb/mongo/commit/30d7a2645b4ef0c3b09e2582120774fcc687407e

Comment by Mathias Stearn [ 30/Mar/11 ]

Not just a display issue (see test)

Comment by auto [ 30/Mar/11 ]

Author:

{u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: Test for SERVER-2828 with broken bits disabled
Branch: master
https://github.com/mongodb/mongo/commit/95c0b3b5cfe834f6f6251adb7c4bb92e09125565

Comment by Eliot Horowitz (Inactive) [ 24/Mar/11 ]

this is just a display issue

Generated at Thu Feb 08 03:01:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.