OplogProviderKEKRotationTest.InitialKmEntryIsUnencryptedPassthrough is racy

XMLWordPrintableJSON

    • Server Security
    • Fully Compatible
    • ALL
    • Server Security 2026-08-28
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      OplogProviderKEKRotationTest.InitialKmEntryIsUnencryptedPassthrough in src/mongo/db/modules/atlas/src/disagg_storage/pali/oplog_provider_test.cpp fails intermittently. Seen on unit_tests / amazon-linux2023-arm64-static-compile in BF-45790:

      src/mongo/db/modules/atlas/src/disagg\_storage/pali/oplog\_provider\_test.cpp:1096: Failure
      Value of: isEncryptedIt\->second.bool\_val\(\)
        Actual: true
      Expected: false
      Initial km entry \(kekId=1\) must be forwarded unencrypted
      

      This is a test bug, not a product bug: the encryption path behaved correctly for the slot state it observed.

      Root Cause

      The test depends on the two-slot per-LSN encryptor state in SLSLogEncryptorManagerImpl:* A fresh manager (generateKEK=true, i.e. _activateKEK with no kmEntryLsn) sets _previousOplogEncryptor to SLSLogEncryptorNoOp and _currentOplogEncryptorLsn to 0.

      • getOplogEncryptorForLsn(lsn) returns the current (real) encryptor when lsn > _currentOplogEncryptorLsn, otherwise the previous slot.
      • OplogProvider::makeLogEntry sets MONGO_D_LOG_METADATA_IS_ENCRYPTED from oplogEncryptor->encryptsData().

      The test body runs in this order:

      logProvider.start() — the tailing thread is already running

      insertKMEntryToOplog(...) — the WUOW commits, so the record can become visible

      rebindCurrentOplogEncryptorLsn(_manager, recordId) — only now does _currentOplogEncryptorLsn move off 0

      Between steps 2 and 3 the provider thread can read the km record while _currentOplogEncryptorLsn is still 0. Then lsn > 0 selects the real KEK-1 encryptor, the entry is encrypted, and IS_ENCRYPTED is true. The later waitForAllEarlierOplogWritesToBeVisible call does not close the window — it forces visibility, it is not what makes visibility possible, since the oplog visibility thread can advance on its own after the commit.

      Scope of Work

      * src/mongo/db/modules/atlas/src/disagg_storage/pali/oplog_provider_test.cpp — in InitialKmEntryIsUnencryptedPassthrough, move logProvider.start() to after the km insert and the rebindCurrentOplogEncryptorLsn call, so the tailing thread cannot observe the km record with a stale slot LSN. Other tests in the file (e.g. WriteConflictsAreRetried) already use insert-then-start, so starting after the insert reads the entry fine.
      * src/mongo/db/modules/atlas/src/disagg_storage/pali/oplog_provider_test.cppKmEntryIsSignedByPreviousKEK has the same start-before-mutate ordering. There a pre-activation read happens to still satisfy the assertions (KEK 1 signing, encrypted), so the race is latent rather than flaky, but it should be reordered the same way.

      Acceptance Criteria

      * Both KEK-rotation tests set up the manager slot state before the provider starts tailing.
      * bazel run +db_pali_test passes; repeated runs of the two KEK-rotation tests are stable.

      Technical Notes

      * No product change is needed. There is no evidence of a KEK-selection defect in getOplogEncryptorForLsn or OplogProvider::makeLogEntry.
      * Not a security issue: assertion failure on log-entry metadata in a unit test, no invariant/crash and no user-controllable input.

            Assignee:
            Sam Frank
            Reporter:
            Ken Martin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: