[DRIVERS-1449] Investigate changes in PM-1897: Faster Initial Sync Method Created: 20/Nov/20  Updated: 27/May/22  Resolved: 20/Nov/20

Status: Closed
Project: Drivers
Component/s: None
Fix Version/s: None

Type: Epic Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Driver Changes: Needed
Server Compat: 5.2
Upstream Changes Summary:

This project adds to the Enterprise Server a new method of adding a node to a replica set, or resyncing a node in a replica set. It is supported only on nodes which are running durable (not inMemory) WiredTiger as the storage engine, and the new node and an existing node (which it syncs from) must have the same WiredTiger configuration in terms of whether they are using the Encrypted Storage Engine, their storage.directoryPerDb setting, and their storage.directoryForIndexes setting.

It will not be the default initial sync method on Enterprise Server. There will be a new parameter, initialSyncMethod, which can be set to "fileCopyBased" to use file copy base initial sync. We expect this new method to be considerably faster, but at least for now it is somewhat more limited. In addition to the storage engine limitations, only one node may be initial syncing (with this new method) from a given existing node at a time. That existing node cannot be undergoing a backup at a time, nor can a backup start on a node which is being synced from.

Also possibly relevant for security, unlike logical initial sync, file copy based initial sync with encrypted storage engine cannot be used to re-key the data.


 Description   
Downstream Change Summary

Preemptively checking off Drivers Attention for this project for the team to review and be aware of.

Description of Linked Ticket

Epic Summary

Summary

Add an option to perform initial sync using file copies.

Motivation

Initial sync takes a long time. It does a logical copy, which will always be slower than solutions that do a physical copy. When doing initial sync, most users care the most about the speed of the initial sync instead of other benefits received from logical copy, such as compaction. Users don’t always have a backup to restore from to do initial sync. We don’t officially support tools/processes for initial sync using iterative rsync or backup cursors. By building this functionality directly into the server, we’ll make this the option of choice for users that don’t need to sync across storage engines or across versions.

Cast of Characters

  • Product Owner:
  • Project Lead:
  • Program Manager:
  • Drivers Contact:

Documentation

Scope Document
Technical Design Document
Product Description


Generated at Thu Feb 08 08:23:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.