[SERVER-73536] Account for the size of the outgoing request in bulkWrite sub-batching logic Created: 02/Feb/23  Updated: 29/Oct/23  Resolved: 25/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Lingzhi Deng Assignee: Kaitlin Mahar
Resolution: Fixed Votes: 0
Labels: milestone-2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-72787 Implement sub-batching logic for bulk... Closed
Related
is related to SERVER-77653 Batch write size estimation logic on ... Open
Assigned Teams:
Replication
Backwards Compatibility: Fully Compatible
Sprint: Repl 2023-02-20, Repl 2023-03-20, Repl 2023-05-15, Repl 2023-05-29, Repl 2023-06-12, Repl 2023-06-26
Participants:

 Description   

When constructing the sub-batch bulkWrite command to send to shards, we should accounts for the size of the outgoing request, similar to this.



 Comments   
Comment by Githook User [ 23/Jun/23 ]

Author:

{'name': 'Kaitlin Mahar', 'email': 'kaitlin.mahar@mongodb.com', 'username': 'kmahar'}

Message: SERVER-73536 Account for the size of the outgoing request in bulkWrite sub-batching logic
Branch: master
https://github.com/mongodb/mongo/commit/d3c6f9a58fc73147ed83dbec0276d5f66f3ffeaa

Comment by Kaitlin Mahar [ 07/Jun/23 ]

On further inspection, we currently always include all the namespaces from the incoming client request in the sub-batches, and don't factor in whether any writes in the sub-batch target each namespace. So instead we can just calculate up front the size of the nsInfo and factor that into our command size estimation.

Per conversation with the QE team, we should also factor in top-level fields in the command which could be quite large, e.g. let (see SERVER-74806). (Relatedly, we do not factor in those top-level fields on mongos for existing batched writes; I filed SERVER-77653 to cover doing that.)

We should also factor in the size of command metadata e.g. $audit but SERVER-53387 already covers fixing that problem for all commands so we will not attempt to tackle it here.

Comment by Kaitlin Mahar [ 26/Apr/23 ]

Wanted to note while I've been looking at this code that making getWriteSizeBytes() work for bulkWrite may be a little tricky. Right now that method only produces estimates that are valid for how inserts, updates, and deletes are represented in the insert/update/delete commands respectively, and the method only has to think about the size of the op itself.
But since bulkWrite allows multiple namespaces, we should probably consider for each operation whether or not the batch already contains an operation with the same namespace. If it does, we only have to think about the size of the op itself; but if it doesn't, then the command size would need to grow by both the size of the op and the size of its namespace.
We could be conservative and just always assume the namespace isn't already present, but I think for workloads where many ops go to the same namespace we could end up significantly underfilling batches with that approach.

Generated at Thu Feb 08 06:24:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.