Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30784

Allow sharded change streams to target just the shards that have chunks

    • Query Optimization

      Currently change streams on sharded collections target all shards in the cluster, regardless of which shards actually have data for that collection. In order for change streams to target just the shards that have chunks, there are a few things we need.
      First we need to be able to reconstruct the routing table at any point in the past, so that when we resume a stream we can know which shards had chunks at the time that the stream is resuming from.
      Second, we need chunk migration commit to be a multi-document transaction. If the donor shard recorded a chunk migration commit operation with the same optime as the chunk migration commit operation on the config server, then when a change stream encounters that commit operation on the donor shard, it could use the multi-version routing table to tell whether the recipient shard had any other chunks for that collection as of the time of the commit and if not it could close its change stream cursor to force mongos to retarget. We could then remove the no-op oplog entry we currently log on the donor shard when we migrate a chunk to a shard that has no chunks for that collection.

            backlog-query-optimization Backlog - Query Optimization
            spencer@mongodb.com Spencer Brody (Inactive)
            2 Vote for this issue
            13 Start watching this issue