Migration recipient corrupts the persisted orphan count by decrementing it for xferMods deletes that remove no document

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 6.0.0, 7.0.0, 8.0.0
    • Component/s: Sharding
    • None
    • Environment:
      Any sharded cluster performing chunk migrations where the donor's delete stream can include an {{_id}} that is not present on the recipient.
    • Cluster Scalability
    • ALL
    • Hide

      Steps to reproduce

      1. Begin a chunk migration for a range, for example X from 100 up to 200.
      2. Arrange for the donor's delete batch to include an _id that the
        recipient does not currently hold. This happens when a document is deleted
        on the donor before it is cloned, or when deferred-update reconciliation
        produces a redundant delete (see Ticket A).
      3. Allow the recipient to apply the batch.
      4. Inspect the persisted orphan count for the range, for example the
        numOrphanDocs value recorded for the range deletion task.

      Expected result

      The orphan count reflects only documents actually removed from the
      recipient.

      Actual result

      The orphan count is decremented once per delete entry regardless of
      whether a document was removed, drifting below the true value and
      potentially going negative.

      Show
      Steps to reproduce Begin a chunk migration for a range, for example X from 100 up to 200. Arrange for the donor's delete batch to include an _id that the recipient does not currently hold. This happens when a document is deleted on the donor before it is cloned, or when deferred-update reconciliation produces a redundant delete (see Ticket A). Allow the recipient to apply the batch. Inspect the persisted orphan count for the range, for example the numOrphanDocs value recorded for the range deletion task. Expected result The orphan count reflects only documents actually removed from the recipient. Actual result The orphan count is decremented once per delete entry regardless of whether a document was removed, drifting below the true value and potentially going negative.
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • 🔵 Done

      Problem

      On the migration recipient, the apply step for xferMods deletes processes
      each _id in the delete batch by issuing a justOne delete and then
      decrementing an in-flight orphan delta. That delta is later persisted and
      feeds the recipient's orphan accounting and range-deleter bookkeeping for
      the migrated range.

      Root cause

      The orphan delta is decremented once per delete entry, unconditionally,
      even when the delete removed nothing:

      writeConflictRetry(opCtx, "transferModsDeletes", _nss, [&] {
          deleteObjects(opCtx, collection, id, true /* justOne */, false /* god */, true /* fromMigrate */);
      });
      
      changeInOrphans--;   // decremented even when no document was removed
      didAnything = true;
      

      The recipient legitimately receives deletes for an _id that it does
      not currently hold. Examples:

      • a document deleted on the donor before the clone scan reached it
      • a redundant or idempotent delete produced by deferred-update
        reconciliation

      In all of these cases the delete matches no document on the recipient,
      yet the orphan count is still decremented. Over time this drives the
      persisted orphan count below the true value and can make it negative,
      corrupting orphan bookkeeping for the range.

      Relationship to the missed-delete bug

      This defect is independent of the missed-delete orphan bug tracked in
      Ticket A and pull request
      mongodb/mongo#1819.

      The fix in Ticket A can legitimately emit an additional idempotent delete
      during deferred reconciliation, which makes this mis-count easier to
      trigger. The two changes should therefore land together.

      Affected code

      • Recipient: MigrationDestinationManager::_applyMigrateOp in
        src/mongo/db/s/migration_destination_manager.cpp

      Proposed fix

      The justOne delete already returns the number of documents removed.
      Decrement the orphan delta only when that count is greater than zero:

      const auto numDeleted = writeConflictRetry(opCtx, "transferModsDeletes", _nss, [&] {
          return deleteObjects(opCtx, collection, id, true, false, true);
      });
      if (numDeleted > 0) {
          changeInOrphans--;
      }
      didAnything = true;
      

      No additional reads are required, and the change is local to the delete
      branch of the apply step.

      Impact and severity

      Medium. The persisted orphan count drifts away from reality, which can
      mislead orphan-related metrics and range-deleter accounting. There is no
      direct user-data corruption, but the bookkeeping inaccuracy is
      operationally confusing and can compound across migrations.

            Assignee:
            Unassigned
            Reporter:
            Seid Muhammed (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: