Feature flag util must retry upon errors in suites with stepdowns and kill primary

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.1.0-rc0, 7.3.3, 8.0.0-rc5, 7.0.11
    • Affects Version/s: 7.0.0, 8.0.0-rc0, 7.3.0
    • Component/s: None
    • None
    • Catalog and Routing
    • Fully Compatible
    • ALL
    • v8.0, v7.3, v7.0
    • 15
    • 2
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      The feature flags test util is offering some functions to retrieve the status of feature flags in sharded clusters.

      Purpose of this ticket is to make the library resilient to failovers since we've been observing some [rare] failures in suites with stepdowns and primary nodes termination.

      Examples of errors hit during testing:

      Error: interrupted at shutdown
[fsm_workload_test:write_without_shard_key_with_resharding] 
_constructFromExistingSeedNode/rst.nodes<@src/mongo/shell/replsettest.js:3968:24
_constructFromExistingSeedNode@src/mongo/shell/replsettest.js:3965:35
ReplSetTest/<@src/mongo/shell/replsettest.js:4023:43
retryOnRetryableError@src/mongo/shell/utils.js:64:20
ReplSetTest@src/mongo/shell/replsettest.js:4019:30
getAllReplicas/replicas<@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:24:52
DBQuery.prototype.map@src/mongo/shell/query.js:473:16
getAllReplicas@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:24:34
getPrimaries@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/fixture_helpers.js:240:16
setConn@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:35:50
getStatus@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:48:24
isPresentAndEnabled@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/libs/feature_flag_util.js:150:16
init@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_workloads/write_without_shard_key_base.js:466:29
init@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js:30:28
runFSM@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_libs/fsm.js:157:20
@eval:8:19
main@/data/mci/33b9787d8cdc4879bb784df855fdb225/src/jstests/concurrency/fsm_libs/worker_thread.js:242:23

      Foreground jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js
Error: socket exception [CONNECT_ERROR] server [Connection handshake failed. Is your mongod/mongos 3.4 or older? :: caused by :: network error while attempting to run command 'hello' on host 'localhost:20005'  :: caused by :: dbclient error communicating with server localhost:20005 :: caused by :: futurize :: caused by :: Connection refused]
[fsm_workload_test:write_without_shard_key_with_resharding] 
Mongo.prototype.runCommand@src/mongo/shell/mongo.js:163:17
runCommandWithRetries/<@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_background_op_in_progress.js:56:24
assert.soon@src/mongo/shell/assert.js:364:21
runCommandWithRetries@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_background_op_in_progress.js:52:12
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runCommandOverrideBody@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/network_error_and_txn_override.js:1008:38
runCommandOverride@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/network_error_and_txn_override.js:1085:43
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runCommandWithRetries/<@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:26:24
assert.soon@src/mongo/shell/assert.js:364:21
runCommandWithRetries@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:23:12
Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/implicitly_retry_on_database_drop_pending.js:131:12
runCommandWithReadAndWriteConcerns@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/set_read_and_write_concerns.js:160:17
overrideRunCommand/Mongo.prototype.runCommand@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/override_methods/override_helpers.js:82:20
runClientFunctionWithRetries@src/mongo/shell/session.js:348:42
runCommand@src/mongo/shell/session.js:465:53
DB.prototype._runCommandImpl@src/mongo/shell/db.js:163:45
DB.prototype.runCommand@src/mongo/shell/db.js:178:21
Mongo.prototype.adminCommand@src/mongo/shell/mongo.js:155:32
getStatus@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/feature_flag_util.js:61:18
isPresentAndEnabled@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/libs/feature_flag_util.js:147:16
init@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_workloads/write_without_shard_key_base.js:449:29
init@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_workloads/write_without_shard_key_with_resharding.js:30:28
runFSM@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_libs/fsm.js:157:20
@eval:8:19
main@/data/mci/996418c3bb92f826703469187e318aaf/src/jstests/concurrency/fsm_libs/worker_thread.js:241:23

            Assignee:
            Tommaso Tocci
            Reporter:
            Pierlauro Sciarelli
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: