[SERVER-78137] HPDM fetcher/storage transaction batching for indexes Created: 15/Jun/23  Updated: 14/Jul/23  Resolved: 14/Jul/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Huayu Ouyang Assignee: Matt Kneiser
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Execution NAMR Team 2023-07-10, Execution NAMR Team 2023-07-24
Participants:

 Description   

2- Investigate to increase the batch size of the storage transaction.

  • Set fetcher batch size - setBatchSize (first batch), setLimit (2nd batch and beyond)
  • See if we still need to further batch the results of the fetcher so that we don't try to do too many collection/index inserts within one WUOW. - we should check that the size of a WUOW batch is < 16MB so that it doesn't go beyond the oplog entry size


 Comments   
Comment by Matt Kneiser [ 14/Jul/23 ]

I observed decreased throughput by aligning fetcher batches with commit batches.

Comment by Matt Kneiser [ 07/Jul/23 ]

I'm seeing quite a bit of batching that we are relying on in onBatchedWriteCommit.

For 75 fetcher batches, there are 375 sub-batches. This is an average of 4.9 sub-batches per batch.

Generated at Thu Feb 08 06:37:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.