-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Execution
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Use a batched direct-CRUD write API to apply container writes in a single storage call per batch, rather than one storage call per key.
While we can already replicate batches of container writes through SERVER-130643, the application side still unpacks every batched entry and applies it key-by-key. applyContainerOperations (src/mongo/db/repl/oplog.cpp:3248-3459) contains four separate per-key loops:
- bytes-key with an array of values, insert – oplog.cpp:3350-3358
- bytes-key with a single or absent value, insert – :3364-3368
- int-key range insert – :3378-3386
- bytes-key range delete – :3438-3443
What is already batched is the transaction, not the storage work: all keys of all grouped operations share one WriteUnitOfWork (oplog.cpp:3287, committed at :3457) and one commit timestamp, and since SERVER-132235 they also share one reused cursor per ident (:3303-3319). What remains is an individual cursor->insert / cursor->remove call per key.
This depends on a batched write entry point being added to KVEngineDirectCrudCursor / storage_engine_direct_crud / WiredTigerKVEngine, which is tracked separately in SERVER-132551.
Note on standbys
The original framing of this ticket described this as batching application "on standby nodes". That is misleading about where the work goes: standbys are not a distinct applier. disagg_storage/oplog_application_coordinator.cpp:238-245 constructs a plain repl::OplogApplierImpl in OplogApplication::Mode::kSecondary; only how entries are delivered differs (pushed into _oplogWriteBuffer by the log server manager, rather than fetched from a sync source). Batching and application then run through the identical applyOplogBatchCommon -> groupAndApplyContainerOps -> applyContainerOperations chain.
So a change in applyContainerOperations covers secondaries and standbys alike. The profiling that motivated this work was done on a standby (see SERVER-132235 and PERF-9239), and measurement of this change should likewise be done on a standby during an index build, but the code path is shared.
One reason batched application is expected to pay off more on the apply side than on the primary: the blind-write policy sampled via KVEngine::chooseBlindWritePolicy (wiredtiger_kv_engine.cpp:489-497) resolves, for disaggregated storage, to "not a primary" (disaggregated_persistence_provider.cpp:228-234), so on any non-primary the batch can be applied blind with no read-before-write per key.
- depends on
-
SERVER-132551 Add a batched write API to the direct-CRUD storage layer
-
- Backlog
-
- is related to
-
SERVER-130643 Support oplog application of batched container writes
-
- Closed
-
-
SERVER-132235 Reuse a cursor for container operations in oplog application
-
- Closed
-
- related to
-
SERVER-132551 Add a batched write API to the direct-CRUD storage layer
-
- Backlog
-