[SERVER-74312] Avoid issuing too many automerge requests in parallel Created: 23/Feb/23  Updated: 07/Mar/23  Resolved: 07/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: [DO NOT USE] Backlog - Sharding EMEA
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-74582 Automerger must not send too many req... Closed
Assigned Teams:
Sharding EMEA
Participants:

 Description   

Currently, from a high level point of view, the auto-merger is:

  1. Constructing a <shard, nss> map (that is being optimized under SERVER-73378)
  2. for each shard:
  3. - For each namespace
  4. ---- Issue an auto merger request (in parallel)
  5. - Wait for previous merge requests to drain before throttling and issuing requests for next shard

In presence of thousands of collections, this may result in performance issues:

  • Sending 100K network requests to the same shard in case of 100K collections with mergeable chunks on the same shard
  • The commit of merges (for different collections) will serialize on the CSRS under the chunks lock

Generated at Thu Feb 08 06:27:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.