-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: 6.0.0, 7.0.0, 8.0.0
-
Component/s: Sharding
-
None
-
Environment:Any sharded cluster performing chunk migrations where the donor's delete stream can include an {{_id}} that is not present on the recipient.
-
Cluster Scalability
-
ALL
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
🔵 Done
Problem
On the migration recipient, the apply step for xferMods deletes processes
each _id in the delete batch by issuing a justOne delete and then
decrementing an in-flight orphan delta. That delta is later persisted and
feeds the recipient's orphan accounting and range-deleter bookkeeping for
the migrated range.
Root cause
The orphan delta is decremented once per delete entry, unconditionally,
even when the delete removed nothing:
writeConflictRetry(opCtx, "transferModsDeletes", _nss, [&] { deleteObjects(opCtx, collection, id, true /* justOne */, false /* god */, true /* fromMigrate */); }); changeInOrphans--; // decremented even when no document was removed didAnything = true;
The recipient legitimately receives deletes for an _id that it does
not currently hold. Examples:
- a document deleted on the donor before the clone scan reached it
- a redundant or idempotent delete produced by deferred-update
reconciliation
In all of these cases the delete matches no document on the recipient,
yet the orphan count is still decremented. Over time this drives the
persisted orphan count below the true value and can make it negative,
corrupting orphan bookkeeping for the range.
Relationship to the missed-delete bug
This defect is independent of the missed-delete orphan bug tracked in
Ticket A and pull request
mongodb/mongo#1819.
The fix in Ticket A can legitimately emit an additional idempotent delete
during deferred reconciliation, which makes this mis-count easier to
trigger. The two changes should therefore land together.
Affected code
- Recipient: MigrationDestinationManager::_applyMigrateOp in
src/mongo/db/s/migration_destination_manager.cpp
Proposed fix
The justOne delete already returns the number of documents removed.
Decrement the orphan delta only when that count is greater than zero:
const auto numDeleted = writeConflictRetry(opCtx, "transferModsDeletes", _nss, [&] { return deleteObjects(opCtx, collection, id, true, false, true); }); if (numDeleted > 0) { changeInOrphans--; } didAnything = true;
No additional reads are required, and the change is local to the delete
branch of the apply step.
Impact and severity
Medium. The persisted orphan count drifts away from reality, which can
mislead orphan-related metrics and range-deleter accounting. There is no
direct user-data corruption, but the bookkeeping inaccuracy is
operationally confusing and can compound across migrations.