Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-60775

PrimaryOnlyService won't wait for prior Instance on step up if InstanceID had completed on an interceding primary already (ABA problem)

    XMLWordPrintable

Details

    • Bug
    • Status: Backlog
    • Major - P3
    • Resolution: Unresolved
    • 5.0.0, 5.1.0-rc0
    • None
    • None
    • ALL
    • 1

    Description

      PrimaryOnlyService::onStepUp() has logic to wait for any Instances constructed in a previous term have finished executing before constructing any new Instances in a higher term.

      // This ensures that all instances from previous term have joined.
      for (auto& instance : savedInstances) {
          instance.second.waitForCompletion();
      }
      

      The logic in PrimaryOnlyService::onStepUp() only applies to Instances which are still tracked in PrimaryOnlyService::_activeInstances. When the state document for the Instance is removed, the Instance is also removed from PrimaryOnlyService::_activeInstances. However, PrimaryOnlyServiceOpObserver::onDelete() also run on secondaries as part of oplog application.

      This leads to a situation where an Instance can be constructed in term 7 despite an (untracked) Instance with a different ID from term 5 not having its future returned by run() become ready. I suspect the solution here is to have PrimaryOnlyServiceOpObserver check whether the current node is primary when doing the delete/drop before removing the ActiveInstance from the map.

       

      Acceptance criteria: Investigate the root cause and propose possible solutions for triage. 

      Attachments

        Issue Links

          Activity

            People

              backlog-server-servicearch Backlog - Service Architecture
              max.hirschhorn@mongodb.com Max Hirschhorn
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: