Details
-
Investigation
-
Resolution: Done
-
Major - P3
-
None
-
None
-
None
-
Not Needed
Description
Details will be hashed out in the scoping and designing
Description of Linked Ticket
Summary
Allow batches of size > 1 for updates that modify a document’s owning shard and improve their latency and throughput.
Motivation
MongoDB 4.2 introduced the ability to atomically change the shard key value of a document. This feature requires the update (or findAndModify) command to be run as a retryable write or within a multi-document transaction. However, it has a limitation where the bulk update command must contain only a single operation. This restricts the valid update commands an application or driver can send to a sharded cluster. In preparation for the world where users no longer choose their own shard key, it must be possible for applications and drivers to send these kinds of update commands to a sharded cluster without error.
Additionally, the current implementation of document shard key modifications is driven entirely by MongoS and as such it has 2 major performance drawbacks:
- The document is transferred between MongoS and the shards 4 times, which results in significant latency when the moves are across geographical zones.
- There can't be more than one update in the batch, because of the need to start and commit a distributed transaction for the move and we only get to use the transaction id of what the driver sent
The ability to update a document’s shard key has spiked a number of interesting use-cases that customers are trying to implement:
- The ability to encode the “hotness” of a document as part of the shard key and to use modifications on that field in order to move documents between hot and cold shards
- The ability to move documents across geographical zones
Documentation
Scope Document
Technical Design Document
Product Description