Use-after-free in __wt_disagg_enqueue_metadata_operation when debug logging is enabled

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 382.375
    • SE Foundations - 2026-09-01
    • None

      Task: disagg_repl_jscore_passthrough on enterprise-amazon-linux2023-arm64-disagg-no-feature-flags-extra-system-deps

      Link: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_amazon_linux2023_arm64_disagg_no_feature_flags_extra_system_deps_display_disagg_repl_jscore_passthrough_patch_30326bf63adf48766d48700131909c9c4acf884f_6a7988ee5d606c0007fd66ef_26_08_10_08_17_05

      wt_disagg_enqueue_metadata_operation() (src/conn/conn_layered.c) inserts a new entry into the shared metadata queue (disaggregated_storage.shared_metadata_qh) under shared_metadata_queue_lock, and then emits __wt_verbose_debug2 messages that dereference the entry (notably entry->stable_value) to format the log output.

      Once inserted, the entry is visible to consumer threads, which dequeue entries via TAILQ_REMOVE and free them (see the consumers around lines ~599, ~624, ~756). There is no synchronization between those consumers and the enqueueing thread's post-insertion debug reads.

      Race window:

      Thread A builds an entry, takes the spinlock, inserts it into the queue, releases the lock.
      Thread B (metadata consumer) dequeues the entry, processes it, frees it.
      Thread A continues into the debug-log block and reads entry->stable_value — now freed memory.
      Trigger conditions:

      Requires verbose debug logging (WT_VERB_DISAGGREGATED_STORAGE at debug2 level) — with logging off, the racy reads never execute, which is likely why this has gone unnoticed.
      Requires a consumer to process and free the entry within the narrow window between enqueue and log formatting.
      Impact: Use-after-free read; garbage in debug output at best, crash under ASan / heap reuse at worst.

      Suggested fix: Move the debug logging to before the queue insertion — the entry is exclusively owned by the enqueueing thread until it is inserted, so logging first is race-free. The "Cannot fail past this point" comment and lock/insert block move down accordingly.

        1. fix.png
          fix.png
          156 kB

            Assignee:
            Salman Javed
            Reporter:
            Salman Javed
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: