[SERVER-44192] Introduce metrics to measure the usefulness of the Top Chunk Optimisation Created: 24/Oct/19 Updated: 06/Dec/22 Resolved: 03/Mar/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kaloian Manassiev | Assignee: | [DO NOT USE] Backlog - Sharding EMEA |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Sharding EMEA
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
The so called "Top Chunk Optimisation" in Sharding operates by checking whether writes are happening to the highest (or lowest) chunk of a collection and pre-splitting and possibly moving that chunk to a different shard. This is done in order to ensure that the "top chunks" are moved while empty (or as small as possible), rather than when they are full of data. Given that it only looks at the extreme chunks of the collection, the top chunk optimisation appears to only be useful for the cases where the collection is bulk-loaded, on a single thread, using data sorted on the shard key. Without this pattern, the top chunk optimisation is largely unused and in certain cases causes issues, because it would block the threads performing inserts to the extreme chunks. The blocking issue was mostly fixed in 4.2 by moving the auto-splitter to the shard servers and making it asynchronous, but in older versions it still persists. This ticket is to design and introduce metrics, which would validate the usefulness of the TCO. If it is determined that it is not useful, we will remove it. |
| Comments |
| Comment by Cris Insignares Cuello [ 03/Mar/22 ] |
|
As of 4.2 with the moving of the auto splitter to the shards, the top chunk optimization no longer blocks incoming rights. As a result, it causes more harm than good. |
| Comment by Sheeri Cabral (Inactive) [ 23/Jan/20 ] |
|
Useful metrics: how long a migration triggered by the top-chunk optimisation took and how much data it moved (size of chunk, either in bytes or # docs). If we see that the size of chunks moved by the TCO is always of size 0, then that's (possibly) a good sign, otherwise it's a waste. kaloian.manassiev - I was thinking for all chunks, so we can measure effectiveness with and without. For example: with TCO: 100 chunks moved, size 0, total migration time 10s without TCO: 10 chunks moved, size 1k (or 100 docs), total migration time 5s There's more overhead with TCO, even though the size is 0. But we need data for all migrations to decide that. We could add in a flag for whether or not a migration was TCO, so we can verify that the TCO migrations are for empty chunks only.
|