Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-12975

Investigate changes in SERVER-42022: Attempt to remove initial sync missing document fetching

      Description

      Downstream Change Summary

      During initial sync, a replica clones all collections, while at the same time buffer its sync source's new oplog entries. Once cloning completes, the replica sets its "minValid" timestamp to the latest oplog entry, and replays the oplog up to minValid. At this point the replica has constructed a consistent snapshot of the data, and it can transition to a secondary.

      In MMAPv1 days, it was possible to miss a document while cloning a collection, even if the document existed during the whole time we were cloning, because the document was moved during the clone. Therefore, during recovery or initial sync, when a replica would apply an update operation that referred to a missing document (either from a regular update or the "applyOps" command) it would attempt to fetch the document from the sync source, in case the document had been missed during cloning. If the document was found, the replica would advance its minValid timestamp to the sync source's current opTime, setting back its progress toward achieving a consistent snapshot.

      In MongoDB 4.3 we no longer support MMAPv1, and we don't even support syncing from a server that uses MMAPv1. Post-MMAPv1 we can't miss documents during collection clone, so it is no longer necessary to fetch missing documents, nor to advance minValid. If an update refers to a missing document, then there must be a later oplog entry that deletes the document, so the replica can skip the update operation.

      We don't have much documentation of this process, so not much has to change. replSetGetStatus.initialSyncStatus.fetchedMissingDocs field is removed, see two references in replSetGetStatus.txt and one in getLog.txt. There may be other places that need updating that I have no found, however!

      Description of Linked Ticket

      We currently think this was only necessary for mmapv1 when updates could cause a collection scan to miss a document. If we could remove this step, it could significantly speed up initial sync in certain workloads and simplify initial sync.

      This would require the storage engine API to prohibit updates from causing collection scans to miss a document.

      Scope of changes

      Impact to Other Docs

      MVP (Work and Date)

      Resources (Scope or Design Docs, Invision, etc.)

            Assignee:
            kay.kim@mongodb.com Kay Kim (Inactive)
            Reporter:
            backlog-server-pm Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:
              4 years, 6 weeks, 3 days ago