[SERVER-28943] Make shards retry non-write commands on stale version exceptions Created: 24/Apr/17 Updated: 30/Oct/23 Resolved: 11/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.2.12, 3.4.4, 3.5.6 |
| Fix Version/s: | 4.9.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Esha Maharishi (Inactive) | Assignee: | Tommaso Tocci |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | PM-1965-Cleanup | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||
| Sprint: | Sharding 2020-11-02, Sharding 2020-11-16 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||||||
| Description |
|
If a shard encounters a shardVersion mismatch in checkShardVersionOrThrow, it returns a stale version error even if the sender was more fresh. When a shard is returning a stale version error, it also refreshes its own routing table just before sending the response. In the case the mongos was more fresh, the mongos is forced to refresh its routing table cache and send the request again. Then, since the shard refreshed just before responding, the shard will accept the request (unless another migration/dropCollection/unshardCollection has happened). This wastes two network round-trips: the mongos has to refresh from the config servers even though it's not stale, and the mongos has to re-send the request to the shard. It would be better if the shard refreshed and re-tried checkShardVersionOrThrow, and only responded with a stale version error if the sender was more stale. This is an improvement/optimization and not that easy of a change, since the collection is locked when checkShardVersionOrThrow is called (so we can't simply move the shard's routing table refresh to checkShardVersionOrThrow). |
| Comments |
| Comment by Githook User [ 11/Nov/20 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Tommaso Tocci [ 02/Nov/20 ] |
|
I'm re-opening this because during skunkwork project I've implemented the retry logic on the shards for non-write commands. |
| Comment by Esha Maharishi (Inactive) [ 01/Nov/20 ] |
|
This is an old cleanup ticket that wouldn't be scheduled independently. It may get done as part of other work. |
| Comment by Dianna Hohensee (Inactive) [ 05/Dec/17 ] |
|
Putting this is PM-675, because this will lead to the mongos needing to re-establish shard point-in-time reads less often – the assumption here is that if a mongos receives a StaleShardVersion error from one shard, it must resend to all shards in order to ensure every shard is using the correct filtering metadata for that point in time. |
| Comment by Githook User [ 15/Jun/17 ] |
|
Author: {u'username': u'EshaMaharishi', u'name': u'Esha Maharishi', u'email': u'esha.maharishi@mongodb.com'}Message: |
| Comment by Esha Maharishi (Inactive) [ 09/Jun/17 ] |
|
Temporary workaround will be to bump the retries from 3 to 10 in scatterGather(). |
| Comment by Esha Maharishi (Inactive) [ 23/May/17 ] |
|
Note, one way to implement this could be to wrap all paths on mongod that call onStaleShardVersion() in a loop, to allow the whole operation to be retried (not just retrying the shardVersion check). This is fairly straightforward (at least code-changes wise; not sure what the side-effects might be, especially for non-idempotent operations) for finds over OP_QUERY and other non-write commands:
Write commands might be more difficult, since onStaleShardVersion() is called in handleError(), which is called in multiple places. |
| Comment by Esha Maharishi (Inactive) [ 23/May/17 ] |
|
Putting this back into Needs Triage, as it is the preferred fix for the linked BF-5311 (see the comment on that BF). |