Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54167

configOpTime can become ahead of VectorClock::clusterTime

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Run the sharding suite with the changes in this patch that adds an invariant:

       

      invariant(configTime.getTimestamp() <= currentTime.clusterTime().asTimestamp());
      

      in Grid.cpp and check that it gets tripped.

       

       

      Show
      Run the sharding suite with the changes in  this patch  that adds an invariant:   invariant(configTime.getTimestamp() <= currentTime.clusterTime().asTimestamp()); in Grid.cpp and check that it gets tripped.    
    • Sprint:
      Sharding 2021-02-08
    • Linked BF Score:
      45

      Description

      The configOpTime (Grid::configOpTime()) can get ahead of the Vector's Clock clusterTime following the this sequence of events:

      1. Node A is sending a request to node B.
      2. In A, when preparing the request metadata, it will first get the vector clock components and add them to the request.
      3. Concurrently, another thread in A bumps the clocks (vector clock & configOpTime). Now configOpTime is greater than the clusterTime that we read at point 2
      4. A adds the configOpTime to the request metadata (which because of point 3, is greater than the clusterTime it wrote at 2
      5. The request gets sent to node B, which updates it's clock with the received times. B is left with a configOpTime greter than VectorClock::clusterTime
      6. When something later calls Grid::configOpTime, the invariant is tripped.

      The order in which the metadata egress hooks are called is set here for the mongos and here for mongod. Changing the hooks order so that the configOpTime hook runs before the vectorClock hook would not solve the issue, because of the fact that the hooks are run on the same order when reading reply metadata. If we did so, then the configOpTime would get advanced before VectorClock, leading to the same situation.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jordi.serra-torrens Jordi Serra Torrens
              Reporter:
              jordi.serra-torrens Jordi Serra Torrens
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: