[SERVER-28943] Make shards retry non-write commands on stale version exceptions Created: 24/Apr/17  Updated: 30/Oct/23  Resolved: 11/Nov/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.2.12, 3.4.4, 3.5.6
Fix Version/s: 4.9.0

Type: Improvement Priority: Major - P3
Reporter: Esha Maharishi (Inactive) Assignee: Tommaso Tocci
Resolution: Fixed Votes: 0
Labels: PM-1965-Cleanup
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
causes SERVER-57051 Shard may fail to notify that router ... Closed
Related
is related to SERVER-29630 bump number of stale version retries ... Closed
Backwards Compatibility: Minor Change
Sprint: Sharding 2020-11-02, Sharding 2020-11-16
Participants:
Linked BF Score: 0

 Description   

If a shard encounters a shardVersion mismatch in checkShardVersionOrThrow, it returns a stale version error even if the sender was more fresh.

When a shard is returning a stale version error, it also refreshes its own routing table just before sending the response.

In the case the mongos was more fresh, the mongos is forced to refresh its routing table cache and send the request again. Then, since the shard refreshed just before responding, the shard will accept the request (unless another migration/dropCollection/unshardCollection has happened).

This wastes two network round-trips: the mongos has to refresh from the config servers even though it's not stale, and the mongos has to re-send the request to the shard.

It would be better if the shard refreshed and re-tried checkShardVersionOrThrow, and only responded with a stale version error if the sender was more stale.

This is an improvement/optimization and not that easy of a change, since the collection is locked when checkShardVersionOrThrow is called (so we can't simply move the shard's routing table refresh to checkShardVersionOrThrow).



 Comments   
Comment by Githook User [ 11/Nov/20 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-28943 Make shards retry non-write commands on stale version exceptions
Branch: master
https://github.com/mongodb/mongo/commit/3eabee611446687824c2560ab78dc51e57b91d75

Comment by Tommaso Tocci [ 02/Nov/20 ]

I'm re-opening this because during skunkwork project I've implemented the retry logic on the shards for non-write commands.

Comment by Esha Maharishi (Inactive) [ 01/Nov/20 ]

This is an old cleanup ticket that wouldn't be scheduled independently. It may get done as part of other work.

Comment by Dianna Hohensee (Inactive) [ 05/Dec/17 ]

Putting this is PM-675, because this will lead to the mongos needing to re-establish shard point-in-time reads less often – the assumption here is that if a mongos receives a StaleShardVersion error from one shard, it must resend to all shards in order to ensure every shard is using the correct filtering metadata for that point in time.

Comment by Githook User [ 15/Jun/17 ]

Author:

{u'username': u'EshaMaharishi', u'name': u'Esha Maharishi', u'email': u'esha.maharishi@mongodb.com'}

Message: SERVER-28943 increase stale version retries from 3 to 10
Branch: master
https://github.com/mongodb/mongo/commit/4af14770aa104cf86592fa97cc9e5878be74b096

Comment by Esha Maharishi (Inactive) [ 09/Jun/17 ]

Temporary workaround will be to bump the retries from 3 to 10 in scatterGather().

Comment by Esha Maharishi (Inactive) [ 23/May/17 ]

Note, one way to implement this could be to wrap all paths on mongod that call onStaleShardVersion() in a loop, to allow the whole operation to be retried (not just retrying the shardVersion check).

This is fairly straightforward (at least code-changes wise; not sure what the side-effects might be, especially for non-idempotent operations) for finds over OP_QUERY and other non-write commands:

  • finds over OP_QUERY: onStaleShardVersion() is called in receivedQuery, and that try-catch could just be put in a loop
  • other non-write commands: onStaleShardVersion() is called in execCommandDatabase, and this try-catch could also be put in a loop, though the try-catch is longer so there might be complications

Write commands might be more difficult, since onStaleShardVersion() is called in handleError(), which is called in multiple places.

Comment by Esha Maharishi (Inactive) [ 23/May/17 ]

Putting this back into Needs Triage, as it is the preferred fix for the linked BF-5311 (see the comment on that BF).

Generated at Thu Feb 08 04:19:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.