Chunk migration recipient retains an orphaned document when a deferred xferMods update moves the document out of range before it is deleted

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: 7.0.0, 6.0.5, 8.0.0
    • Component/s: Sharding
    • None
    • Environment:
      Sharded cluster running multi-document or retryable transactions with
      replica-set failovers (term changes) concurrent with the balancer,
      moveChunk, or moveRange.
    • Cluster Scalability
    • ALL
    • Hide

      Steps to reproduce

      1. Set up a sharded cluster, shard a collection on the key X, and begin
        migrating the chunk range from X equal to 100 up to X equal to 200
        from a donor shard to a recipient shard.
      2. While the migration is in its catch-up phase, have a transaction that
        was prepared in a previous replica-set term commit an update on the
        document with _id equal to X and shard key X equal to 150,
        changing its shard key to an out-of-range value such as X equal to 90.
        Inducing a donor primary step-down and step-up makes the prepared
        transaction survive the term change, so the commit oplog entry has no
        post-image document key and the update is deferred.
      3. Delete the same document, which now sits at X equal to 90, outside
        the migrating range. The donor drops this delete as out of range.
      4. Allow the recipient to pull the next batch of changes, which runs the
        deferred reconciliation step.
      5. Allow the migration to complete.

      Expected result

      The document is absent on the recipient because it was deleted on the
      donor.

      Actual result

      The recipient retains the document: an orphaned and effectively
      resurrected record.

      A deterministic unit-level reproduction is provided by the test
      DeferredUpdateForRemovedInRangeDocModelsDelete in
      migration_chunk_cloner_source_test.cpp.

      Show
      Steps to reproduce Set up a sharded cluster, shard a collection on the key X , and begin migrating the chunk range from X equal to 100 up to X equal to 200 from a donor shard to a recipient shard. While the migration is in its catch-up phase, have a transaction that was prepared in a previous replica-set term commit an update on the document with _id equal to X and shard key X equal to 150, changing its shard key to an out-of-range value such as X equal to 90. Inducing a donor primary step-down and step-up makes the prepared transaction survive the term change, so the commit oplog entry has no post-image document key and the update is deferred. Delete the same document, which now sits at X equal to 90, outside the migrating range. The donor drops this delete as out of range. Allow the recipient to pull the next batch of changes, which runs the deferred reconciliation step. Allow the migration to complete. Expected result The document is absent on the recipient because it was deleted on the donor. Actual result The recipient retains the document: an orphaned and effectively resurrected record. A deterministic unit-level reproduction is provided by the test DeferredUpdateForRemovedInRangeDocModelsDelete in migration_chunk_cloner_source_test.cpp .
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • 🔵 Done

      sent to the recipient, even though the recipient
      already received the document while its pre-image was still in range.
      After the migration commits and the recipient takes ownership of the
      range, the recipient holds a document that no longer exists on the donor.
      In effect, a deleted document is silently resurrected.

      Affected code

      • Donor reconciliation: MigrationChunkClonerSource::_processDeferredXferMods
        in src/mongo/db/s/migration_chunk_cloner_source.cpp
      • Out-of-range delete drop: MigrationChunkClonerSource::onDeleteOp in
        the same file

      Proposed fix

      In the "document not found" branch of the deferred reconciliation,
      extract the shard key from the deferred pre-image document key. If the
      pre-image fell inside the chunk range, enqueue a delete for that
      _id to the recipient.

      The delete is precise, because nothing is sent for documents the
      recipient never received, and it is idempotent, because a duplicate
      delete by _id simply removes nothing on the recipient.

      Impact and severity

      High. This is a silent data-integrity problem for sharded clusters that
      combine prepared transactions, failovers (term changes), and chunk
      migrations. The resulting orphaned documents can reappear in queries
      routed to the new owning shard and can violate unique constraints.

      See the pull request for the implementation:
      mongodb/mongo#1819

            Assignee:
            Unassigned
            Reporter:
            Seid Muhammed (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: