Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-1954

SDAM should give priority to electionId over setVersion when updating topology

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • None
    • SDAM
    • None
    • Needed
    • Hide

      Filed DRIVERS-1954 to track drivers change

      Show
      Filed DRIVERS-1954 to track drivers change
    • Hide
      • Sync spec changes in 5bd06a8
      • Confirm tests fail with current updateRSFromPrimary implementation
      • Update updateRSFromPrimary to
        • Prioritize electionId before setVersion
        • Handle nullish values for both setVersion and electionId
        • always set maxElectionId and maxSetVersion together (they're a tuple value)
      • Confirm tests pass with changes
      • Update April 1 2022: Minor fixes were made, notably the spec files were using hello, where it should have been helloOk, see commit: 316c650 to pull in the latest.
      Show
      Sync spec changes in 5bd06a8 Confirm tests fail with current updateRSFromPrimary implementation Update updateRSFromPrimary to Prioritize electionId before setVersion Handle nullish values for both setVersion and electionId always set maxElectionId and maxSetVersion together (they're a tuple value) Confirm tests pass with changes Update April 1 2022: Minor fixes were made, notably the spec files were using hello, where it should have been helloOk, see commit: 316c650 to pull in the latest.

    Description

      in progress...

      Summary

      SDAM spec specifies that RSM is using the { setVersion, electionId } in that order to detect stale primaries. The motivation for this is that if the protocol version changes (like it happened in 3.2.0) the electionId might not be directly comparable but the setVersion is guaranteed to increment. Details: https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#using-setversion-and-electionid-to-detect-stale-primaries 

      The problem with that if the failover happens before the former primary was able to get the consensus on setVersion increment the new primary will communicate a decremented setVersion while electionId incremented. The existing SDAM treats this as stale primary, which leads to full cluster outage and requires manual intervention. Details in SERVER-59409.

      Drawback: if we need to make non-compatible protocol versions in future, which will make the electionId non monotonical, if will require an additional contingency plan.

      Tests: the SDAM updated in head to match new behavior: https://github.com/mongodb/mongo/tree/master/src/mongo/client/sdam/json_tests/sdam_tests 

      Motivation

      Who is the affected end user?

      Who are the stakeholders? Divers team, server teams.

      How does this affect the end user?

      Full cluster outage is possible.

      How likely is it that this problem or use case will occur?

      It happens in tests all the time.

      If the problem does occur, what are the consequences and how severe are they?

      Outage.

      Is this issue urgent?

      Not urgent but high priority.

      Is this ticket required by a downstream team?

      TBD, might be just normal upgrade path.

      Is this ticket only for tests?

      No.

      Attachments

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. SERVER-59409 Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary

          • Major - P3 - Major loss of function.
          • Closed

          Task - A task that needs to be done. DRIVERS-2196 Sync SDAM tests from mongo server repository

          • Major - P3 - Major loss of function.
          • Closed
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. DRIVERS-2412 SDAM should prioritize electionId over setVersion only on >=6.0 servers

          • Unknown - Priority not determined yet.
          • Implementing
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. DRIVERS-2412 SDAM should prioritize electionId over setVersion only on >=6.0 servers

          • Unknown - Priority not determined yet.
          • Implementing
          split to

          Bug - A problem which impairs or prevents the functions of the product. CDRIVER-4203 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. CSHARP-3934 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. CXX-2404 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. GODRIVER-2207 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MOTOR-847 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NODE-3712 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. PHPC-2068 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. PYTHON-2970 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. RUBY-2829 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. RUST-1081 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SWIFT-1400 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. JAVA-4375 SDAM should give priority to electionId over setVersion when updating topology

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              neal.beeken@mongodb.com Neal Beeken
              andrew.shuvalov@mongodb.com Andrew Shuvalov (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: