[SERVER-49787] Create collection bulk loader for resharding Created: 21/Jul/20  Updated: 29/Oct/23  Resolved: 04/Nov/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: PM-234-M2, PM-234-T-data-clone
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-49293 Test collection bulk loader for resha... Closed
Gantt Dependency
has to be done after SERVER-49785 Write and test aggregation pipeline f... Closed
Related
related to SERVER-52690 Switch ReshardingCollectionCloner to ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2020-09-21, Sharding 2020-10-05, Sharding 2020-10-19, Sharding 2020-11-16
Participants:

 Description   

The collection bulk loader for resharding should take the aggregation pipeline from SERVER-49785, run it against a particular donor, and insert the returned documents into the new sharded collection on itself (a recipient).

  • Serialize the aggregation pipeline from createCloningPipelineForResharding() using Pipeline::serializeToBson() and use the serialized form to create an AggregationRequest.
  • Add the necessary read concern and hint to the AggregationRequest.
  • Run the aggregate command on the remote donor. Unclear if this should be using ClusterAggregate::runAggregate() or some other mechanism. sharded_agg_helpers::attachCursorToPipeline() has come up as a way to having the merging happen on the local node (the recipient in the case of this ticket) for computing the new initial split; see also the changes from 262e5a9 as part of SERVER-49525.
  • Use Collection::insertDocuments() to insert the documents into the new sharded collection.


 Comments   
Comment by Githook User [ 04/Nov/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-49787 Create collection cloner for resharding.
Branch: master
https://github.com/mongodb/mongo/commit/ff6685cf6fa936ab8b8769f496e5a6e35dba228e

Generated at Thu Feb 08 05:20:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.