[SERVER-44192] Introduce metrics to measure the usefulness of the Top Chunk Optimisation Created: 24/Oct/19  Updated: 06/Dec/22  Resolved: 03/Mar/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: [DO NOT USE] Backlog - Sharding EMEA
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-44193 Remove Top Chunk Optimisation Closed
Related
is related to SERVER-61557 Get rid of top chunk migration Closed
Assigned Teams:
Sharding EMEA
Participants:

 Description   

The so called "Top Chunk Optimisation" in Sharding operates by checking whether writes are happening to the highest (or lowest) chunk of a collection and pre-splitting and possibly moving that chunk to a different shard. This is done in order to ensure that the "top chunks" are moved while empty (or as small as possible), rather than when they are full of data.

Given that it only looks at the extreme chunks of the collection, the top chunk optimisation appears to only be useful for the cases where the collection is bulk-loaded, on a single thread, using data sorted on the shard key.

Without this pattern, the top chunk optimisation is largely unused and in certain cases causes issues, because it would block the threads performing inserts to the extreme chunks. The blocking issue was mostly fixed in 4.2 by moving the auto-splitter to the shard servers and making it asynchronous, but in older versions it still persists.

This ticket is to design and introduce metrics, which would validate the usefulness of the TCO. If it is determined that it is not useful, we will remove it.



 Comments   
Comment by Cris Insignares Cuello [ 03/Mar/22 ]

As of 4.2 with the moving of the auto splitter to the shards, the top chunk optimization no longer blocks incoming rights. As a result, it causes more harm than good.

Comment by Sheeri Cabral (Inactive) [ 23/Jan/20 ]

Useful metrics: how long a migration triggered by the top-chunk optimisation took and how much data it moved (size of chunk, either in bytes or # docs). If we see that the size of chunks moved by the TCO is always of size 0, then that's (possibly) a good sign, otherwise it's a waste.

kaloian.manassiev - I was thinking for all chunks, so we can measure effectiveness with and without. For example:

with TCO: 100 chunks moved, size 0, total migration time 10s

without TCO: 10 chunks moved, size 1k (or 100 docs), total migration time 5s

There's more overhead with TCO, even though the size is 0. But we need data for all migrations to decide that.

We could add in a flag for whether or not a migration was TCO, so we can verify that the TCO migrations are for empty chunks only.

 

Generated at Thu Feb 08 05:05:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.