Uploaded image for project: 'Compass '
  1. Compass
  2. COMPASS-6212

Investigate changes in PM-2015: Remove restrictions and increase performance of shard key updates

    XMLWordPrintableJSON

Details

    • Icon: Investigation Investigation
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • No version
    • None
    • None
    • None
    • Not Needed

    Description

      Original Downstream Change Summary

      Details will be hashed out in the scoping and designing

      Description of Linked Ticket

      Epic Summary

      Summary

      Allow batches of size > 1 for updates that modify a document’s owning shard and improve their latency and throughput.

      Motivation

      MongoDB 4.2 introduced the ability to atomically change the shard key value of a document. This feature requires the update (or findAndModify) command to be run as a retryable write or within a multi-document transaction. However, it has a limitation where the bulk update command must contain only a single operation. This restricts the valid update commands an application or driver can send to a sharded cluster. In preparation for the world where users no longer choose their own shard key, it must be possible for applications and drivers to send these kinds of update commands to a sharded cluster without error.

      Additionally, the current implementation of document shard key modifications is driven entirely by MongoS and as such it has 2 major performance drawbacks:

      • The document is transferred between MongoS and the shards 4 times, which results in significant latency when the moves are across geographical zones.
      • There can't be more than one update in the batch, because of the need to start and commit a distributed transaction for the move and we only get to use the transaction id of what the driver sent

      The ability to update a document’s shard key has spiked a number of interesting use-cases that customers are trying to implement:

      • The ability to encode the “hotness” of a document as part of the shard key and to use modifications on that field in order to move documents between hot and cold shards
      • The ability to move documents across geographical zones

      Documentation

      Scope Document
      Technical Design Document
      Product Description

      Attachments

        Activity

          People

            Unassigned Unassigned
            backlog-server-pm Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: