Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-89943

Feature flag util must retry upon errors in suites with stepdowns and kill primary

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.1.0-rc0, 7.3.3, 8.0.0-rc5, 7.0.11
    • Affects Version/s: 7.0.0, 8.0.0-rc0, 7.3.0
    • Component/s: None
    • None
    • Catalog and Routing
    • Fully Compatible
    • ALL
    • v8.0, v7.3, v7.0
    • 15
    • 2

      The feature flags test util is offering some functions to retrieve the status of feature flags in sharded clusters.

      Purpose of this ticket is to make the library resilient to failovers since we've been observing some [rare] failures in suites with stepdowns and primary nodes termination.

      Examples of errors hit during testing:

      Error: interrupted at shutdown
[fsm_workload_test:write_without_shard_key_with_resharding] 
_constructFromExistingSeedNode/rst.nodes<@src/mongo/shell/replsettest.js:3968:24
_constructFromExistingSeedNode@src/mongo/shell/replsettest.js:3965:35
ReplSetTest/<@src/mongo/shell/replsettest.js:4023:43
retryOnRetryableError@src/mongo/shell/utils.js:64:20
ReplSetTest@src/mongo/shell/replsettest.js:4019:30
getAllReplicas/replicas<@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:24:52
DBQuery.prototype.map@src/mongo/shell/query.js:473:16
getAllReplicas@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:24:34
getPrimaries@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:240:16
setConn@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:35:50
getStatus@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:48:24
isPresentAndEnabled@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:150:16
init@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_workloads/write_without_shard_key_base.js:466:29
init@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js:30:28
runFSM@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_libs/fsm.js:157:20
@eval:8:19
main@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_libs/worker_thread.js:242:23

      Foreground jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js
Error: socket exception [CONNECT_ERROR] server [Connection handshake failed. Is your mongod/mongos 3.4 or older? :: caused by :: network error while attempting to run command 'hello' on host 'localhost:20005'  :: caused by :: dbclient error communicating with server localhost:20005 :: caused by :: futurize :: caused by :: Connection refused]
[fsm_workload_test:write_without_shard_key_with_resharding] 
Mongo.prototype.runCommand@src/mongo/shell/mongo.js:163:17
runCommandWithRetries/<@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_background_op_in_progress.js:56:24
assert.soon@src/mongo/shell/assert.js:364:21
runCommandWithRetries@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_background_op_in_progress.js:52:12
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runCommandOverrideBody@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/network_error_and_txn_override.js:1008:38
runCommandOverride@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/network_error_and_txn_override.js:1085:43
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runCommandWithRetries/<@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:26:24
assert.soon@src/mongo/shell/assert.js:364:21
runCommandWithRetries@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:23:12
Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:131:12
runCommandWithReadAndWriteConcerns@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/set_read_and_write_concerns.js:160:17
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runClientFunctionWithRetries@src/mongo/shell/session.js:348:42
runCommand@src/mongo/shell/session.js:465:53
DB.prototype._runCommandImpl@src/mongo/shell/db.js:163:45
DB.prototype.runCommand@src/mongo/shell/db.js:178:21
Mongo.prototype.adminCommand@src/mongo/shell/mongo.js:155:32
getStatus@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/feature_flag_util.js:61:18
isPresentAndEnabled@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/feature_flag_util.js:147:16
init@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_workloads/write_without_shard_key_base.js:449:29
init@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js:30:28
runFSM@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_libs/fsm.js:157:20
@eval:8:19
main@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_libs/worker_thread.js:241:23

            Assignee:
            tommaso.tocci@mongodb.com Tommaso Tocci
            Reporter:
            pierlauro.sciarelli@mongodb.com Pierlauro Sciarelli
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: