[COMPASS-6212] Investigate changes in PM-2015: Remove restrictions and increase performance of shard key updates Created: 17/Oct/22  Updated: 19/Oct/22  Resolved: 19/Oct/22

Status: Closed
Project: Compass
Component/s: None
Affects Version/s: None
Fix Version/s: No version

Type: Investigation Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documentation Changes: Not Needed

 Description   
Original Downstream Change Summary

Details will be hashed out in the scoping and designing

Description of Linked Ticket

Epic Summary

Summary

Allow batches of size > 1 for updates that modify a document’s owning shard and improve their latency and throughput.

Motivation

MongoDB 4.2 introduced the ability to atomically change the shard key value of a document. This feature requires the update (or findAndModify) command to be run as a retryable write or within a multi-document transaction. However, it has a limitation where the bulk update command must contain only a single operation. This restricts the valid update commands an application or driver can send to a sharded cluster. In preparation for the world where users no longer choose their own shard key, it must be possible for applications and drivers to send these kinds of update commands to a sharded cluster without error.

Additionally, the current implementation of document shard key modifications is driven entirely by MongoS and as such it has 2 major performance drawbacks:

  • The document is transferred between MongoS and the shards 4 times, which results in significant latency when the moves are across geographical zones.
  • There can't be more than one update in the batch, because of the need to start and commit a distributed transaction for the move and we only get to use the transaction id of what the driver sent

The ability to update a document’s shard key has spiked a number of interesting use-cases that customers are trying to implement:

  • The ability to encode the “hotness” of a document as part of the shard key and to use modifications on that field in order to move documents between hot and cold shards
  • The ability to move documents across geographical zones

Documentation

Scope Document
Technical Design Document
Product Description


Generated at Wed Feb 07 22:42:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.