[SERVER-78253] Allow folks with hashed shard keys to drop the hashed index Created: 20/Jun/23  Updated: 29/Oct/23  Resolved: 02/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0, 7.0.2, 5.0.22, 6.0.11

Type: Improvement Priority: Major - P3
Reporter: Garaudy Etienne Assignee: Ali Mir
Resolution: Fixed Votes: 0
Labels: sharding-nyc-subteam3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-78995 Allow users to hide the last compatib... Closed
Problem/Incident
Related
related to SERVER-78863 Audit places in the codebase that rel... Closed
related to SERVER-78252 Block chunk migrations for hashed sha... Closed
Assigned Teams:
Sharding NYC
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.0, v6.0, v5.0
Sprint: Sharding NYC 2023-07-10, Sharding NYC 2023-07-24, Sharding NYC 2023-08-07
Participants:
Linked BF Score: 160
Story Points: 2

 Description   

The hashed index is only needed to support chunk migrations. However, it hurts write performance. Most collections with hashed shard keys have even data distribution and growth, thus not requiring chunk migrations. As such, we should allow customers to drop the hashed index for hashed shard keys. 



 Comments   
Comment by Githook User [ 06/Oct/23 ]

Author:

{'name': 'Sanika Phanse', 'email': 'sanika.phanse@mongodb.com', 'username': 'sphanse99'}

Message: SERVER-78253 Allow folks with hashed shard keys to drop the hashed index
Branch: v5.0
https://github.com/mongodb/mongo/commit/7307701f41f431d927d0385f02b8d335a0df475c

Comment by Githook User [ 26/Sep/23 ]

Author:

{'name': 'Sanika Phanse', 'email': 'sanika.phanse@mongodb.com', 'username': 'sphanse99'}

Message: SERVER-78253 Allow folks with hashed shard keys to drop the hashed index
Branch: v6.0
https://github.com/mongodb/mongo/commit/40e3a1a11c73783b62c972ebc8183f3ff0debde9

Comment by Githook User [ 18/Sep/23 ]

Author:

{'name': 'Sanika Phanse', 'email': 'sanika.phanse@mongodb.com', 'username': 'sphanse99'}

Message: SERVER-78253 Allow folks with hashed shard keys to drop the hashed index
Branch: v7.0
https://github.com/mongodb/mongo/commit/1a3b5d60e2f6a9b53f65c0bde9c1b7cd01b5f214

Comment by Githook User [ 22/Aug/23 ]

Author:

{'name': 'Ben Shteinfeld', 'email': 'ben.shteinfeld@mongodb.com', 'username': 'bshteinfeld'}

Message: SERVER-79853 [CQF] Correctly handle hashed shard keys when performing shard filtering

When a shard key contains a hashed component, the ShardFilterer expects
us to pass it a value which is also hashed. We achieve this by inserting
an invocation of the 'shardHash' SBE builtin function where necessary.

This patch also updates the CQF fallback mechanism to enable CQF for
hashed shard keys when a natural hint is provided and/or there is no
hashed index backing the shard key (this is allowed by SERVER-78253).
Branch: master
https://github.com/mongodb/mongo/commit/694d0aae817b891a3752a9900309c98ce471c5a8

Comment by Githook User [ 15/Aug/23 ]

Author:

{'name': 'Ben Shteinfeld', 'email': 'ben.shteinfeld@mongodb.com', 'username': 'bshteinfeld'}

Message: SERVER-79991 Fallback to classic when we encounter a collection with a
hashed shard key.

SERVER-78253 changed the behavior of dropIndexes() to drop hashed
indexes which are backing hashed shard keys. The fallback mechanism was
assuming that a hashed shard key will always have a backing hashed
index; now this assumption is no longer valid. This change explicitly
falls back when encountering a hashed shard key to account for cases
where the backing index is dropped.
Branch: master
https://github.com/mongodb/mongo/commit/e081fbfffed44b1db793ec412264d89be91223e1

Comment by Githook User [ 03/Aug/23 ]

Author:

{'name': 'Ali Mir', 'email': 'ali.mir@mongodb.com', 'username': 'ali-mir'}

Message: SERVER-78253 Allow users to drop hashed shard key indexes
Branch: minh.luu-no_compile_sys-perf
https://github.com/mongodb/mongo/commit/a37e840f44677193f67dfafae26e14357db743b0

Comment by Githook User [ 02/Aug/23 ]

Author:

{'name': 'Ali Mir', 'email': 'ali.mir@mongodb.com', 'username': 'ali-mir'}

Message: SERVER-78253 Allow users to drop hashed shard key indexes
Branch: master
https://github.com/mongodb/mongo/commit/a37e840f44677193f67dfafae26e14357db743b0

Comment by Githook User [ 25/Jul/23 ]

Author:

{'name': 'Sviatlana Zuiko', 'email': 'sviatlana.zuiko@mongodb.com', 'username': 'szuiko'}

Message: Revert "SERVER-78253 Allow users to drop hashed shard key indexes"

This reverts commit e3d2bc5d42088a9752ac37a3076f3b4e0b98d05a.
Branch: master
https://github.com/mongodb/mongo/commit/344a6c8e0f735b500b0d8d2e6301742bd5aea239

Comment by Githook User [ 24/Jul/23 ]

Author:

{'name': 'Ali Mir', 'email': 'ali.mir@mongodb.com', 'username': 'ali-mir'}

Message: SERVER-78253 Allow users to drop hashed shard key indexes
Branch: master
https://github.com/mongodb/mongo/commit/e3d2bc5d42088a9752ac37a3076f3b4e0b98d05a

Comment by Garaudy Etienne [ 28/Jun/23 ]

For now let's keep the hashed index and just tell people to keep dropping the hashed index. I want us to think through how to get resharding to drop it a little longer...
 

Comment by Max Hirschhorn [ 28/Jun/23 ]

We still need a "non-hashed" shard key index to be available, right?

randolph@mongodb.com for what purpose other than the client application's own reads and writes does this index need to be available?

Edit: I think I understand what you're getting at now. If we drop the hashed index without having a non-hashed index then point queries which could have previously used the hashed index will now collection scan if there isn't a non-hashed index available.

Comment by Garaudy Etienne [ 28/Jun/23 ]

yes we still need a non-hashed index for everyone with a ranged shard key. 

 

shardCollection and refine don't automatically create the index so they're fine. but yes resharding should stop creating the hash index. 

Comment by Randolph Tan [ 28/Jun/23 ]

We still need a "non-hashed" shard key index to be available, right? Should we also change/add option for shardCollection/refine/reshard to not create the hash index?

Generated at Thu Feb 08 06:37:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.