[SERVER-59023] Resharding can fail with NamespaceNotSharded following a primary failover on a recipient shard Created: 02/Aug/21  Updated: 29/Oct/23  Resolved: 30/Aug/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 5.0.0
Fix Version/s: 5.0.4, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: PM-234-M3, PM-234-T-oplog-apply
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-53351 Add resharding fuzzer task with step-... Closed
Problem/Incident
is caused by SERVER-54984 Relax invariant in ReshardingOpObserv... Closed
Related
related to SERVER-60094 ReshardingOplogApplicationRules does ... Closed
related to SERVER-62178 Resharding can fail with NamespaceNot... Closed
is related to SERVER-52639 Fail resharding operation if new shar... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Sharding 2021-08-09, Sharding 2021-08-23
Participants:
Story Points: 2

 Description   

A secondary which steps up to be primary won't necessarily have loaded the collection metadata for temporary resharding collection. This can lead assertCanExtractShardKeyFromDocs() to throw a NamespaceNotSharded exception when the ReshardingOplogApplier attempts to write to the temporary resharding collection.

The ReshardingCollectionCloner and ReshardingOplogApplier rely on assertCanExtractShardKeyFromDocs() to ensure an update doesn't write an invalid shard key value (e.g. an array value) under the new shard key pattern. We must either

  • (a) ensure the collection metadata for the temporary resharding collection has been loaded before the ReshardingCollectionCloner or ReshardingOplogApplier attempt to write to it, or
  • (b) retry in ReshardingCollectionCloner and ReshardingOplogApplier on the exception thrown by assertCanExtractShardKeyFromDocs(), or
  • (c) move the checks into ReshardingCollectionCloner and ReshardingOplogApplier directly, and allow unversioned (direct) writes to the temporary resharding collection if they are being performed by an internal (system) Client.

const auto metadata = CollectionShardingRuntime::get(opCtx, nss)->getCurrentMetadataIfKnown();
// A user can manually create a 'db.system.resharding.' collection that isn't guaranteed to be
// sharded outside of running reshardCollection.
uassert(ErrorCodes::NamespaceNotSharded,
        str::stream() << "Temporary resharding collection " << nss.toString()
                      << " is not sharded",
        metadata && metadata->isSharded());



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 21/Sep/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-59023 Version internal writes to resharding temp ns as IGNORED.

Adds a retry once on StaleConfig exception helper to enable the
collection metadata for the temporary resharding collection to be
recovered from the config server.

(cherry picked from commit 7cfe35e4384af3c25aef161ab4c8133c50474395)
Branch: v5.0
https://github.com/mongodb/mongo/commit/e9c533f56c2f8c78ac6a0caa704fb45b0c27a2ab

Comment by Githook User [ 30/Aug/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-59023 Version internal writes to resharding temp ns as IGNORED.

Adds a retry once on StaleConfig exception helper to enable the
collection metadata for the temporary resharding collection to be
recovered from the config server.
Branch: master
https://github.com/mongodb/mongo/commit/7cfe35e4384af3c25aef161ab4c8133c50474395

Comment by Max Hirschhorn [ 09/Aug/21 ]

I discussed this ticket with Kal and Tommaso. Kal didn't want to introduce a new, named error code and pointed out that IGNORED is sufficient for ensuring the collection metadata is known by the shard. The plan is to do the following:

  • Call initializeClientRoutingVersion() with IGNORED for the temporary resharding collection.
  • Create a new staleConfigRetry() loop that catches StaleConfig exceptions, calls onShardVersionMismatch(), and retries. This is akin to the logic which lives in service_entry_point_common.cpp already.
  • Put staleConfigRetry() in ReshardingCollectionCloner and ReshardingOplogApplier so the collection metadata is guaranteed to be known in the OpObserver for those writes.
Generated at Thu Feb 08 05:46:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.