[SERVER-30953] Add auto-retry logic to the mongo shell for testing during stepdown suites Created: 05/Sep/17  Updated: 30/Oct/23  Resolved: 21/Sep/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jack Mulrow
Resolution: Fixed Votes: 0
Labels: sharding36-passthrough-testing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-31194 Add a version of retryable_writes_jsc... Closed
Related
related to SERVER-31355 Increase replica set size and stepdow... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2017-10-02
Participants:

 Description   

When a network connection to the mongo shell is closed, attempting to call Mongo.prototype.runCommand() and Mongo.prototype.runCommandWithMetadata() throw a JavaScript exception. We should add a new override in the jstests/libs/override_methods/ directory to catch these exceptions (i.e. ones where isNetworkError() return true) and automatically re-send the command request to the server. The goal of this override is to implement retry logic such that the assertions within our existing JavaScript tests still pass despite stepdowns of the CSRS and replica set shards happening in the background.

The approach for implementing this override can likely be adapted from SessionAwareClient#runClientFunctionWithRetries() with the following changes:

  • If the command request is an "insert", "update", "delete", or "findAndModify" command (or a wrapped query/$query object for one of those commands) for which ServerSession#canRetryWrites()) returns true, then the operation should be retried through the existing retryWrites=true logic in SessionAwareClient.
  • If the command request is an "insert", "update", or "delete" command (or a wrapped query/$query object for one of those command) for which ServerSession#canRetryWrites() returns false, then an exception should be thrown saying something along the lines of "Cowardly refusing to run a test that issues non-retryable write operations since the test likely makes assertions on the write results and can lead to spurious failures if a network error occurs."
  • If the command request is a "getMore", then an exception should be thrown saying something along the lines of "Cowardly refusing to run a test that issues a getMore command since if a network error occurs during it then we won't know whether the cursor was advanced or not."


 Comments   
Comment by Githook User [ 21/Sep/17 ]

Author:

{'email': 'jack.mulrow@mongodb.com', 'name': 'Jack Mulrow', 'username': 'jsmulrow'}

Message: SERVER-30953 Add auto-retry logic to the mongo shell for testing during stepdown suites
Branch: master
https://github.com/mongodb/mongo/commit/442aab2feb4fdaf185bbecf26e0655369ff6262b

Comment by Max Hirschhorn [ 06/Sep/17 ]

kaloian.manassiev, greg.mckeon, let me know if neither Jack nor someone else on the Sharding team has bandwidth to pick this up in the next iteration. My understanding is that it is a prerequisite for being able to have shard stepdowns in the jsCore, concurrency, and fuzzer passthrough suites.

Generated at Thu Feb 08 04:25:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.