[SERVER-47371] Chunk migration concurrent with multi-delete can cause matching documents to not be deleted Created: 06/Apr/20  Updated: 12/Dec/23

Status: Backlog
Project: Core Server
Component/s: Querying, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kevin Pulo Assignee: Backlog - Cluster Scalability
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-46211 Chunk migration concurrent with multi... Closed
Assigned Teams:
Cluster Scalability
Operating System: ALL
Participants:

 Description   

Observed in mr_output_options.js where coll.remove({}) completed successfully, but then coll.find().itcount() != 0 (with no concurrent inserts). This is in contrast to running coll.remove({}) on a standalone or replica set, or on a sharded cluster in the absence of a concurrent chunk migration.

Sequence of events is:

  1. Start chunk migration from shardA to shardB.

  2. After the range deletion on the recipient (shardB), but before the clone starts, the mongos gets coll.remove({}), and broadcasts it unversioned to both shards.

  3. shardB finishes that deletion quickly. shardB now has 0 documents in coll.

  4. Meanwhile, shardA has started processing the multi-delete, but is working on other documents, not those in the chunk range being moved.

  5. Now the clone of documents from shardA to shardB happens (starts and completes). shardB now has non-zero documents (the contents of the chunk being moved).

  6. The migration enters the critical section to commit, interrupting the multi-delete on shardA with StaleConfig "migration commit in progress for dbname.collname".

  7. The migration gets the final xfermods from the donor's OpObsever inside the critical section, but because the multi-delete on shardA hasn't yet gotten to any of the chunk range documents, there are no mods to apply. The migration finishes normally.

  8. In the meantime, the mongos received StaleConfig from the multi-delete on shardA, so it has resent the multi-delete but only to shardA. It blocks until the critical section exits, then runs normally to successful completion. The mongos multi-delete command now also completes successfully. shardA now has 0 documents, but shardB still has the documents from the migrated chunk.


 Comments   
Comment by Alexey Maltsev [ 11/Mar/21 ]

Hello!
We're facing similar issue when updating shard key values in document. First during bulk replace some of the documents are failed to replace, because this update will cause document's migration to another shard. Then we retry all failed replace operation with replaceOne command and sometimes they may also fail with error "Update operation was converted into a distributed transaction because the document being updated would move shards and that transaction failed. :: caused by :: Encountered error from <host>:<port> during a transaction :: caused by :: migration commit in progress for <dbname>.<collname>".

Is there any workaround for the last mentioned error? I saw that in commit for parent issue that balancer was turned off, is it a valid fix?

We're using mongo 4.4 with 3 shards, each consists of 3 mongod.

Generated at Thu Feb 08 05:13:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.