ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Replication
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      SERVER-134162 fixed a bug where a DSC primary's skip-self check in _startHeartbeats() compared each member's _id to the node's array index in the config, assuming that the member's ID and index position would match. After a reconfig reordered the members, the primary skipped the other electable node and heartbeated itself instead. This ocurred silently, as a node that heartbeats itself gets a well-formed response back, so the heartbeat chain keeps running and the bug is invisible from inside the server.

      The impact of this class of bug is that the primary never heartbeats another valid member, and therefore never updates its view of that member's optime. replSetGetStatus reports that member as infinitely lagged, Atlas health checks and plan moves block on it, and the starved standby keeps trying to step up because it never hears from the primary. In HELP-99085 this surfaced only because Atlas noticed the lag; our tests did not, since they do not check replication lag nor do they reorder member config lists.

      A node should never heartbeat itself, so we should detect when this happens and most likely fail rather than continue silently.

      Note that this does not affect ASC. The classic repl coordinator's skip-self check compares array index to _selfIndex, which is resolved dynamically by a node on every config install, so it does not depend on member ordering.

            Assignee:
            Unassigned
            Reporter:
            Ali Mir
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: