[SERVER-49891] Write and test aggregation pipeline for oplog fetcher for resharding Created: 25/Jul/20  Updated: 29/Oct/23  Resolved: 16/Sep/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.8.0

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: PM-234-M2, PM-234-T-oplog-fetch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-49892 Continued testing for aggregation pip... Closed
Gantt Dependency
has to be done before SERVER-49893 Create oplog fetcher for resharding Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2020-09-21
Participants:

 Description   

The goal of this ticket is to create a function which allows the aggregation pipeline to easily be sent to a remote (donor) shard and for the aggregation pipeline to be unit-testable with DocumentSourceMock. It is preferable to use DocumentSourceXX::create() functions (or DocumentSourceXX::parseFromBSON() when the former isn't available or is too tedious) rather than building it up with string concatenation. DocumentSources can be conditionally added to the Pipeline::SourceContainer, for example, to reflect a stage being added only when resuming on a new cursor.

std::unique_ptr<Pipeline, PipelineDeleter> createOplogFetchingPipelineForResharding(
    ReshardingDonorOplogId startAfter,  /* isNull() when not resuming */
    ShardId recipientShard
);

Some of these parameters are probably more appropriate to take by const-ref because their contents can only be copied into the Pipeline anyway.


Testing should cover the following types of oplog entries:

  • Ordinary and retryable insert, update, or delete.
    • Includes formats for $set/$unset updates, replacement updates, and $v:2 delta updates (compressed pipeline updates).
  • The preImageOpTime and postImageOpTime documents from retryable findAndModify operations.
  • The final oplog entry marker from when the donor shard entered the "all new incoming writes as distributed transactions" state.
  • Drop collection.
  • All of the above for the matching (namespace, recipient), a different recipient, and a different namespace.


 Comments   
Comment by Githook User [ 16/Sep/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-49891: Create and test the resharding oplog fetching aggregation pipeline.
Branch: master
https://github.com/mongodb/mongo/commit/b0f66da822b0771a7a00539fc5935647740425ca

Generated at Thu Feb 08 05:21:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.