[SERVER-68687] Update Global Index fetcher pipeline Created: 09/Aug/22  Updated: 17/Nov/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Randolph Tan Assignee: Backlog - Cluster Scalability
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-68685 Add new $_internalIndexKey agg expres... Closed
depends on SERVER-68686 Add new $_internalOwningShard agg exp... Closed
Assigned Teams:
Cluster Scalability
Participants:

 Description   

Pipeline should look like:

[
{
  // only if resuming
  $match: {
    $expr: {$gte: ["$_id", {$literal: <last seen _id>}]}} 
  }
},
{
  $sort: {_id: 1}
},
{
  // Use replaceRoot to guarantee the order of output fields.
  $repacleRoot: {
    newRoot: {
      _id: { 
        $indexKey: {doc: "$$ROOT", spec: <object containing global index spec>}
      },
      documentKey: {
 
        // Use arrayToObject as a work around to output valid shard key patterns with
        // dotted field names.
        $arrayToObject: [[
          {k: "_id", v: "$_id"},
 
          // Explicitly output null value for missing fields, otherwise $arrayToObject complains:
          // "$arrayToObject requires an object keys of 'k' and 'v'. Found incorrect number of keys:1"
          {k: <sourceShardKey1>, v: {$ifNull: ["$sourceShardKey1", null]}},
        ]]
      }
  }
},
{
  // unwind is needed because $indexKey outputs an array to handle multikey 
  $unwind: "$_id"
},
{
  $project: {
    _id: 0,
    shard: {
       $owningShard: {shardKeyVal: {_id: "$_id"}, ns: <globalIndexNs>, uuid: <globalIndexUUID>},
    },
    indexData: "$$ROOT"
  }
},
{
  $match: {shard: <recipientShardId>}
},
{
  $replaceRoot: {
    newRoot: "$indexData"
  }
}
]


Generated at Thu Feb 08 06:11:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.