[SERVER-32198] Missing collection metadata on the shard implies both UNSHARDED and "metadata not loaded yet" Created: 06/Dec/17  Updated: 30/Oct/23  Resolved: 30/Jun/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.4.10, 3.6.0, 4.0.4
Fix Version/s: 4.7.0

Type: Bug Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Fixed Votes: 4
Labels: PM-1645-Milestone-1, ShardingRoughEdges
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ssv_after_restart_of_shards_and_mongos_workaround.js    
Issue Links:
Depends
depends on SERVER-32367 AutoGetCollectionOrView and its relat... Closed
depends on SERVER-35773 MetadataManager directly calls into t... Closed
depends on SERVER-46679 Transaction test fails with Stale Sha... Closed
depends on SERVER-47233 WriteOp can be left in pending state,... Closed
depends on SERVER-31056 Remove all usages of the default cons... Closed
depends on SERVER-35497 Remove usages of OldClient(Write)Cont... Closed
depends on SERVER-35516 Get rid of OldClientWriteContext Closed
depends on SERVER-35691 Make all code paths returning StaleCo... Closed
depends on SERVER-45599 Backport the changes to CollectionSha... Closed
depends on SERVER-45778 Rename getOrphanFilter to getOwnershi... Closed
depends on SERVER-45779 Throw Stale Shard Version when collec... Closed
depends on SERVER-45780 Rename getCurrentMetadata to getColle... Closed
depends on SERVER-45781 Throw Stale Shard Version when collec... Closed
depends on SERVER-45904 Create a standalone collection shardi... Closed
depends on SERVER-46272 Make callers of keyBelongsToMe() to n... Closed
depends on SERVER-46703 Make a variant of checkShardVersionOr... Closed
depends on SERVER-47104 Make upsert stage use OwnershipFilter... Closed
depends on SERVER-47412 Make the database cloner ensure colle... Closed
depends on SERVER-47413 Some transaction tests are not being ... Closed
depends on SERVER-47419 Find a better way of setting the shar... Closed
depends on SERVER-47474 CheckShardingIndex: Convert usages of... Closed
depends on SERVER-47521 Remove getCollectionDescription_DEPRE... Closed
depends on SERVER-47555 Remove checkShardVersionOrThrow_DEPRE... Closed
depends on SERVER-47701 Ensure the steps for upgrading Replic... Closed
depends on SERVER-47982 Change the shard version update proce... Closed
depends on SERVER-47987 Remove forceShardFilteringMetadataRef... Closed
depends on SERVER-36054 Get rid of ScopedCollectionMetadata's... Closed
depends on SERVER-36164 Decouple ScopedCollectionMetadata fro... Closed
Duplicate
is duplicated by SERVER-32459 mongos incorrectly assumed collection... Closed
is duplicated by SERVER-35932 Mongos saved docs in wrong shard Closed
is duplicated by SERVER-24960 Shard may not realize that collection... Closed
is duplicated by SERVER-32075 Load sharding metadata from disk on m... Closed
is duplicated by SERVER-42658 secondary don't refresh its routing ... Closed
is duplicated by SERVER-44052 Inconsistencies in sharded collections Closed
Problem/Incident
causes SERVER-38794 CollectionOptions can be lost in crea... Closed
causes SERVER-38953 Missing early return in merge_chunks_... Closed
Related
related to SERVER-52991 Complete TODO listed in SERVER-32198 Closed
related to SERVER-49228 Remove kickback to mongos requirement... Closed
is related to SERVER-32075 Load sharding metadata from disk on m... Closed
Assigned Teams:
Sharding
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sharding 2018-01-01, Sharding 2017-12-18, Sharding 2018-02-12, Sharding 2018-02-26, Sharding 2018-06-18, Sharding 2018-07-02, Sharding 2018-07-16, Sharding 2018-07-30, Sharding 2018-08-13, Sharding 2018-08-27, Sharding 2018-09-10
Participants:
Case:

 Description   

This ticket was created as an overflow of SERVER-31056. Please refer to the comments there for more context.

If after a migration commit on the config server, a donor shard fails to refresh its metadata, we clear the MetadataManager's metadata. We do this so that the next time the metadata is needed, a refresh will be triggered.

However, if a request comes in with an UNSHARDED shardVersion, it will see empty metadata and the request will be allowed to go through without triggering a refresh.

The linked SERVER-24960 contains test cases to validate the fix.



 Comments   
Comment by Githook User [ 08/Feb/22 ]

Author:

{'name': 'Allison Easton', 'email': 'allison.easton@mongodb.com', 'username': 'allisoneaston'}

Message: SERVER-52991 Complete TODO listed in SERVER-32198
Branch: master
https://github.com/mongodb/mongo/commit/ecae6131ec395e927fcea7c81378fca6fe0862e8

Comment by Kaloian Manassiev [ 30/Jun/20 ]

This work has completed as a result of the commits made to this ticket and all the completed dependencies, which constitute Milestone 1 of PM-1645.

Comment by Githook User [ 26/Feb/20 ]

Author:

{'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-45599 Backport of SERVER-32198: Split CollectionShardingState::getMetadata into three methods

o getCurrentMetadataIfKnown - which returns the current filtering metadata if any is available
o getMetadataForOperation - which returns the metadata which is required by the current opertion, based on the OperationShardingState
o getCurrentMetadata - which returns the currently available filtering metadata (or UNSHARDED if not known)

This is in preparation for making getMetadataForOperation/getCurrentMetadata throw StaleShardVersion exception if the metadata has not been loaded yet.

This is a partial cherry-pick from dcf7e0dd89d34f58b592f1adb3d41e5edd6e2012, adapted for the 4.0 branch.
Branch: v4.0
https://github.com/mongodb/mongo/commit/b41cd4b2c416e965fc3541c97cd0b51563d90c40

Comment by Githook User [ 26/Feb/20 ]

Author:

{'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-45599 Backport of SERVER-32198: Make MetadataManager support an 'UNKNOWN' filtering metadata state

This is partial cherry-pick of the changes done under 20117b8ee9678794be675eb4d728bfcc8f9d75f4 to make the 4.0 logic match that of 4.2.
Branch: v4.0
https://github.com/mongodb/mongo/commit/96ced970aef7c5dbf81b660009bb80f33d8026e8

Comment by Githook User [ 29/Jan/20 ]

Author:

{'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}

Message: Revert "SERVER-45599 Backport of SERVER-32198: Make MetadataManager support an 'UNKNOWN' filtering metadata state"

This reverts commit e4e052a10484cf2cc44c62c377db4ece8162820a.
Branch: v4.0
https://github.com/mongodb/mongo/commit/132aa852b1ae87f48165dbdf50943f32d2376c73

Comment by Githook User [ 29/Jan/20 ]

Author:

{'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}

Message: Revert "SERVER-45599 Backport of SERVER-32198: Split CollectionShardingState::getMetadata into three methods"

This reverts commit a414e4ceafb45dc6ebf4daeb9198f0a7f3fb189c.
Branch: v4.0
https://github.com/mongodb/mongo/commit/c25809db3532d2af31321648790e775bd420e600

Comment by Githook User [ 25/Jan/20 ]

Author:

{'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm', 'name': 'Kaloian Manassiev'}

Message: SERVER-45599 Backport of SERVER-32198: Split CollectionShardingState::getMetadata into three methods

o getCurrentMetadataIfKnown - which returns the current filtering
metadata if any is available
o getMetadataForOperation - which returns the metadata which is required
by the current opertion, based on the OperationShardingState
o getCurrentMetadata - which returns the currently available filtering
metadata (or UNSHARDED if not known)

This is in preparation for making
getMetadataForOperation/getCurrentMetadata throw
StaleShardVersion exception if the metadata has not been loaded yet.

This is a partial cherry-pick from dcf7e0dd89d34f58b592f1adb3d41e5edd6e2012, adapted for the 4.0 branch.
Branch: v4.0
https://github.com/mongodb/mongo/commit/a414e4ceafb45dc6ebf4daeb9198f0a7f3fb189c

Comment by Githook User [ 24/Jan/20 ]

Author:

{'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-45599 Backport of SERVER-32198: Make MetadataManager support an 'UNKNOWN' filtering metadata state

This is partial cherry-pick of the changes done under 20117b8ee9678794be675eb4d728bfcc8f9d75f4 to make the 4.0 logic match that of 4.2.
Branch: v4.0
https://github.com/mongodb/mongo/commit/e4e052a10484cf2cc44c62c377db4ece8162820a

Comment by Githook User [ 14/Nov/18 ]

Warning: Do not backport the commit seen below without also including a bugfix from SERVER-38794.

Author:

{'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}

Message: SERVER-32198 Split CollectionShardingState::getMetadata into three methods

o getCurrentMetadataIfKnown - which returns the current filtering
metadata if any is available
o getMetadataForOperation - which returns the metadata which is required
by the current opertion, based on the OperationShardingState
o getCurrentMetadata - which returns the currently available filtering
metadata (or UNSHARDED if not known)

This is in preparation for making
getMetadataForOperation/getCurrentMetadata throw
StaleShardVersion exception if the metadata has not been loaded yet.
Branch: master
https://github.com/mongodb/mongo/commit/dcf7e0dd89d34f58b592f1adb3d41e5edd6e2012

Comment by Githook User [ 19/Sep/18 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-32198 Get rid of CollectionShardingState::collectionIsSharded

(cherry picked from commit 80de0da37b00dbeed576b28a842cb172b6714358)

SERVER-35773 Remove references to the CatalogCache from MetadataManager

(cherry picked from commit 2aa65a86193e5d38934a4f2d6b0a8298e2432485)

SERVER-32198 Add support for an optional `vWanted` to StaleConfigInfo

(cherry picked from commit 60559a00b81293184922b3418a8e56610edf8dd9)

SERVER-36054 Remove ScopedCollectionMetadata's operator bool

(cherry picked from commit c9c340ad6e9e1f33cb001a8375c62d6b16138c74)

SERVER-36054 Remove more unused methods from CSS/MetadataManager

(cherry picked from commit ca04f5bcf9bfa73c9162b3a77225c997c6deec8a)

SERVER-36116 Get rid of CollectionShardingState::resetAll

(cherry picked from commit db1cc80d13d203b6351f5510f7756cc1c7bfc0ea)

SERVER-36054 Get rid of unused methods from CollectionShardingState

(cherry picked from commit 884d232473dca72e0872f0e540d4c3108c1e0b3d)

SERVER-36164 Decouple ScopedCollectionMetadata from MetadataManager

(cherry picked from commit d91262c4a2ed7d94923c3b1c5ff5d208aa981c73)

SERVER-29908 Move CollectionShardingState under sharding_api_d

(cherry picked from commit e491e284e8066929c8272c96a3128241ab481be8)

SERVER-29908 Remove ShardingState::appendInfo

Expose the ShardingState properties and move the appendInfo logic to be
entirely inside the 'getShardingState' function, which is its only
consumer.

(cherry picked from commit 24e411d5cd7f64c5b4da25a351529cd1873284b8)

SERVER-29908 Move 'updateConfigServerOpTimeFromMetadata' out of ShardingState

(cherry picked from commit 7a97557ce5bf74dc2b663762b7a5ffb9c958d580)

SERVER-29908 Move all runtime logic out of ShardingState

... and move it into a ShardingInitializationMongoD class, which is
responsible for driving the sharding-awareness of the node and setting
it onto ShardingState.

Also gets rid of the 'sharding' library, so there is no more library
dependency cycle.

(cherry picked from commit 200c3dc58410d8b3287a2075cc9b2ad085100e83)

SERVER-29908 Fold the 'sharding_connection_hook' library into 'sharding_initialization'

... and also remove dependency of MongoS on the replication coordinator

(cherry picked from commit fab6864f4edcae7bb304f79e601f1f62cc376a77)
Branch: v4.0
https://github.com/mongodb/mongo/commit/27e0275301eed05bea3d65c766dbe76ee1da9b8a

Comment by Githook User [ 12/Sep/18 ]

Author:

{'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}

Message: SERVER-32198 Do not try to access routing info history for unsharded collections
Branch: master
https://github.com/mongodb/mongo/commit/c055154b81b4baa72f23c6760ace606d80aeeea9

Comment by Githook User [ 11/Sep/18 ]

Author:

{'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}

Message: SERVER-32198 Make MetadataManager support an 'UNKNOWN' filtering metadata state
Branch: master
https://github.com/mongodb/mongo/commit/20117b8ee9678794be675eb4d728bfcc8f9d75f4

Comment by Githook User [ 27/Jun/18 ]

Author:

{'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-32198 Add support for an optional `vWanted` to StaleConfigInfo
Branch: master
https://github.com/mongodb/mongo/commit/60559a00b81293184922b3418a8e56610edf8dd9

Comment by Githook User [ 12/Jun/18 ]

Author:

{'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-32198 Retry on StaleShardVersion errors during collection drop
Branch: master
https://github.com/mongodb/mongo/commit/4b7c76e8f6623070795cb97ed1c93e96e92415ca

Comment by Githook User [ 07/Jun/18 ]

Author:

{'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-32198 Get rid of CollectionShardingState::collectionIsSharded
Branch: master
https://github.com/mongodb/mongo/commit/80de0da37b00dbeed576b28a842cb172b6714358

Comment by Githook User [ 24/May/18 ]

Author:

{'username': 'kaloianm', 'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-32198 Move the sessions collection namespace constant to NamespaceString
Branch: master
https://github.com/mongodb/mongo/commit/e55bf669726e91b21b45964b33f6cced1dcc515f

Comment by Githook User [ 26/Apr/18 ]

Author:

{'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm', 'name': 'Kaloian Manassiev'}

Message: SERVER-32198 Change the namespaces stored as StringData to NamespaceString

This avoids having to cast them to NamespaceString for passing to
functions or to std::string for comparisons.
Branch: master
https://github.com/mongodb/mongo/commit/347e5a9d6340b4eb7a59955a4fa265b918913d5b

Comment by Githook User [ 09/Mar/18 ]

Author:

{'email': 'kaloian.manassiev@mongodb.com', 'name': 'Kaloian Manassiev', 'username': 'kaloianm'}

Message: SERVER-32198 Make ShardedConnectionInfo::getVersion return boost::optional

This allows the logic for making the shard version decision to be
concentrated in CollectionShardingState entirely.
Branch: master
https://github.com/mongodb/mongo/commit/b9e20190b647fea262a8f4e154bbf18d9934a3ba

Comment by Nicholas Zolnierz [ 13/Dec/17 ]

Note that this comment refers to the previous ticket number, but should be updated by this ticket.

Generated at Thu Feb 08 04:29:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.