CheckMultikeyConsistencyInBackground hook reports false multikey divergence on the legacy (non-replicated-multikeyness) path

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The new CheckMultikeyConsistencyInBackground hook (added in SERVER-128070) fails with a "multikey divergence (regular index)" assertion when featureFlagReplicateMultikeynessInTransactions is disabled (e.g. in multiversion_last_patch_old_new suites where FCV is pinned to the older release). The divergence is a one-tick timestamp artifact of the legacy multikey-replication path, not real catalog corruption.

      [js_test:bulk_write_insert:CheckMultikeyConsistencyInBackground] uncaught exception: Error:
      assert failed : multikey divergence (regular index) : {
        ns: "basic_insert.basic",
        indexName: "__safeContent___1",
        atClusterTime: Timestamp(1782143946, 28),
        host_ref:   "localhost:21076", state_ref:   { multikey: true,  multikeyPaths: { __safeContent__: BinData(0, 01) } },
        host_other: "localhost:21077", state_other: { multikey: false, multikeyPaths: { __safeContent__: BinData(0, 00) } }
      }
      

      Root cause. On the legacy path (replicateMultikeyness == false in IndexCatalogEntryImpl::setMultikey), the multikey state is not carried in the oplog. The primary sets the index multikey inside a side transaction and emits only a placeholder noop ("Setting index to multikey"). The two nodes therefore stamp the multikey transition at different timestamps:

      • Primary: sets multikey at the noop's timestamp Timestamp(...,28), which is ordered before the document insert.
      • Secondary: the noop carries no metadata, so the secondary only sets multikey while applying the data insert at Timestamp(...,29) via the MultikeyPathTracker (flushed at the batch timestamp).

      A snapshot read at exactly T = 28 (the hook reads at primary.lastApplied, which landed on the noop) therefore legitimately sees multikey:true on the primary and multikey:false on the secondary. This cross-node timestamp inconsistency is the known limitation that featureFlagReplicateMultikeynessInTransactions was created to fix; the hook's invariant ("multikey state is identical at every clusterTime across members") only holds when that flag is enabled.

      Suggested fix. Make the hook aware of featureFlagReplicateMultikeynessInTransactions / FCV and skip (or relax) the regular-index multikey comparison when multikeyness replication is not in effect, since cross-node timestamp consistency is not guaranteed on the legacy path.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: