Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30953

Add auto-retry logic to the mongo shell for testing during stepdown suites

    • Fully Compatible
    • Sharding 2017-10-02

      When a network connection to the mongo shell is closed, attempting to call Mongo.prototype.runCommand() and Mongo.prototype.runCommandWithMetadata() throw a JavaScript exception. We should add a new override in the jstests/libs/override_methods/ directory to catch these exceptions (i.e. ones where isNetworkError() return true) and automatically re-send the command request to the server. The goal of this override is to implement retry logic such that the assertions within our existing JavaScript tests still pass despite stepdowns of the CSRS and replica set shards happening in the background.

      The approach for implementing this override can likely be adapted from SessionAwareClient#runClientFunctionWithRetries() with the following changes:

      • If the command request is an "insert", "update", "delete", or "findAndModify" command (or a wrapped query/$query object for one of those commands) for which ServerSession#canRetryWrites()) returns true, then the operation should be retried through the existing retryWrites=true logic in SessionAwareClient.
      • If the command request is an "insert", "update", or "delete" command (or a wrapped query/$query object for one of those command) for which ServerSession#canRetryWrites() returns false, then an exception should be thrown saying something along the lines of "Cowardly refusing to run a test that issues non-retryable write operations since the test likely makes assertions on the write results and can lead to spurious failures if a network error occurs."
      • If the command request is a "getMore", then an exception should be thrown saying something along the lines of "Cowardly refusing to run a test that issues a getMore command since if a network error occurs during it then we won't know whether the cursor was advanced or not."

            Assignee:
            jack.mulrow@mongodb.com Jack Mulrow
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: