[SERVER-14506] special top chunk logic can move max chunk to a shard with incompatible tag Created: 09/Jul/14  Updated: 12/May/15  Resolved: 08/Aug/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 2.6.6, 2.7.5

Type: Bug Priority: Major - P3
Reporter: Asya Kamsky Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-14667 mongos does not move top chunk after ... Closed
Duplicate
is duplicated by SERVER-4687 need to check draining and potentiall... Closed
Related
related to SERVER-18361 Sharding "top" splits should respect ... Closed
Tested
tested by SERVER-15831 Enhanced top chunk auto split tests Closed
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of Aug 08, 2014

ISSUE SUMMARY
On tag-aware sharded clusters, when the balancer is active and the top/bottom chunk needs to be split, the new chunk may be temporarily moved to a shard with an incompatible tag thus temporarily violating tagging.

This issue does not apply to sharded clusters using hashed shard keys.

USER IMPACT
Users with a heavy insert load may experience poor performance due to unneccessary migrations.

WORKAROUNDS
Users may consider adding a split point at an extremely high, unreachable shard key to prevent this behavior.

AFFECTED VERSIONS
MongoDB production releases up to 2.6.5 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.6 production release.

Original description

In special logic for splitting top or bottom chunk https://github.com/mongodb/mongo/blob/master/src/mongo/s/d_split.cpp#L886 sets shouldMigrate to true.

https://github.com/mongodb/mongo/blob/master/src/mongo/s/chunk.cpp#L466 goes along with the idea as long as balancing isn't disabled, and uses Shard::pick() to find the new target shard for the "hot" chunk.

However, https://github.com/mongodb/mongo/blob/master/src/mongo/s/shard.cpp#L425 does not seem to take tags into account and ends up moving the chunk to a shard from which it must then immediately be moved in the next round (violating tagging in the meanwhile).

This problem does not apply to hashed shard key since special logic is bypassed in that case.



 Comments   
Comment by Githook User [ 25/Nov/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14506 special top chunk logic can move max chunk to a shard with incompatible tag

Fix test
Branch: v2.6
https://github.com/mongodb/mongo/commit/d0cd0e0b62a85fb47b8be39e945715438410efec

Comment by Githook User [ 25/Nov/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14506 special top chunk logic can move max chunk to a shard with incompatible tag
(cherry picked from commit 9d1e5f95a2739a7672b697deb7d1da04ca8c4117)

Conflicts:
jstests/sharding/top_chunk_autosplit.js
src/mongo/SConscript
src/mongo/s/balance.cpp
src/mongo/s/balancer_policy.cpp
src/mongo/s/chunk.cpp
src/mongo/s/d_split.cpp
Branch: v2.6
https://github.com/mongodb/mongo/commit/fd8a2779971d095b2d13fb4c2dd06a11118908a4

Comment by Githook User [ 25/Nov/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14506 special top chunk logic can move max chunk to a shard with incompatible tag

Make DistributionStatus store ChunkTypes instead of BSONObj
(cherry picked from commit c30b981260e26dfeff315b766e2887d4f7404f89)

Conflicts:
src/mongo/s/balancer_policy.h
Branch: v2.6
https://github.com/mongodb/mongo/commit/a12f25486ecd7888e90950423e249a273d36aed8

Comment by Githook User [ 08/Aug/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14506 special top chunk logic can move max chunk to a shard with incompatible tag
Branch: master
https://github.com/mongodb/mongo/commit/9d1e5f95a2739a7672b697deb7d1da04ca8c4117

Comment by Githook User [ 08/Aug/14 ]

Author:

{u'username': u'renctan', u'name': u'Randolph Tan', u'email': u'randolph@10gen.com'}

Message: SERVER-14506 special top chunk logic can move max chunk to a shard with incompatible tag

Make DistributionStatus store ChunkTypes instead of BSONObj
Branch: master
https://github.com/mongodb/mongo/commit/c30b981260e26dfeff315b766e2887d4f7404f89

Comment by Greg Studer [ 09/Jul/14 ]

A workaround might be to add a split point at an extremely high, unreachable shard key value to prevent the top chunk optimization from coming into effect?

Generated at Thu Feb 08 03:35:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.