[SERVER-53539] TypeCollectionReshardingFields are incorrect following a shard version refresh Created: 30/Dec/20 Updated: 29/Oct/23 Resolved: 18/Mar/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 4.9.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Jordi Serra Torrens |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | PM-234-Catalog-Work, PM-234-M2, PM-234-T-lifecycle | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Description |
|
The collection version is seen by both shards d20020 and d20022 as being 2|1||5fec046614ff529dbac7fa05. However, only shard d20022 correctly sees the coordinator state as "cloning" while shard d20020 incorrectly sees the coordinator state as "preparing-to-donate". This causes the d20020 shard to skip constructing a RecipientStateMachine but leads the coordinator (config server) to believe the d20020 shard has finished refreshing. The resharding operation is then left unable to make further progress. This issue appears to only happen (and only very rarely happen) when the temporary resharding collection is being queried via mongos by the test client. I wonder if there's another issue along the lines of
(These logs are from a patch build where the "Ignoring shard version change" and "Creating recipient state machine" messages have been added.) |
| Comments |
| Comment by Githook User [ 19/Mar/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Jordi Serra Torrens', 'email': 'jordi.serra-torrens@mongodb.com', 'username': 'jordist'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Jordi Serra Torrens [ 02/Mar/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
max.hirschhorn I've filed If, in addition to that, we ensure that for every modification to 'allowMigrations' and 'reshardingFields' on config.collections we bump the collection version (bump one chunk), then we don't need to change the collection epoch. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 16/Feb/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kaloian.manassiev, jordi.serra-torrens, should we be having the change to the allowMigrations setting and the reshardingFields (each time) entail bumping the collection epoch then? Triggering a full refresh is more expensive but would at least guarantee the config.collections metadata is associated with the correct chunk version. I'd like to do the work to make resharding be correct (but perhaps not efficient) if we don't plan on scheduling work now to do something more efficient. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Blake Oler [ 16/Feb/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
jordi.serra-torrens reached out to me to update that the ticket we were depending on to fix this issue ( | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 31/Dec/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I ran another patch build with some additional log messages (namely this one) and found that the "refreshedCollectionVersion" isn't consistent with the "refreshedReshardingFields". The config server is responding back with newer chunks than the associated collection metadata.
The issue is pretty clear to me from looking at getChangedChunks() in config_server_catalog_cache_loader.cpp. The shard issues two separate reads with "majority" read concern level which makes the following scenario possible:
I think the reason why this hasn't historically been an issue is that the collection metadata (collection UUID, shard key pattern, collection default collation) have never been allowed to change after they've been initially set unless the epoch has also changed (where the collection was dropped, re-created, and sharded again). If we wanted to continue to bump the chunk version instead of the epoch, then I could imagine for resharding that we could include the collection version in the donor fields (for the existing sharded collection) and in the recipient fields (for the temporary resharding collection) and change createConfigDiffQuery() to also do {$lte: <collection version>}. This wouldn't help the allowMigrations setting in the config.collections document which is meant to also be used by the Sharded collections support all DDL operations project. I would want to get some input from the Sharding EMEA folks here. CC kaloian.manassiev, tommaso.tocci Steps to reproduce
|