• Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Replication, Replication - Oplog and Recovery
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      SERVER-137256 wrapped the asCluster()'s (jstests/libs/replsettest.js) connectionStatus call in retryOnNetworkError(..., ReplSetTest.kDefaultRetries, 1000), so suites that kill or step down nodes survive a dropped connection to a node that comes back.

       

      BF-47145 showed a side effect. ReplSetTest.waitForState's inner _waitForIndicator call is meant to recover from errors. It probes all nodes with _callHello and picks a live one. It then runs a retry loop against the same connection and never re-probes. If that connection is to a node that is dying and won't return, all 240 retries run (about 13 minutes in the BF logs), exceeding waitForState's 600s timeout. 

      In stepdown_fassert_on_explicit_overridden_rstl_timeout.js, waitForState(DOWN) was being called on a node that had just stepped down. If the node's shut down process and waitForState call overlapped, this node could be picked by _waitForIndicator. _waitForIndicator would then stall on the permanently down node. BF-47145 was fixed in stepdown_fassert_on_explicit_overridden_rstl_timeout.js by waiting for the primary to stop accepting connections before waitForState(DOWN).

       

      Acceptance Criteria: 

      Decide whether asCluster()'s retry should be changed so that other tests which poll a node while it crashes or shuts down cannot hit the same stall.

            Assignee:
            Unassigned
            Reporter:
            Ananya Patil
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: