waitWhileFailPointEnabled wrongly matches comment-less operations against a comment-scoped failpoint, causing an unrecoverable hang

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.0.0-rc3
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • Fully Compatible
    • v9.0
    • QE 2026-08-03, QE 2026-08-17
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      jstests/core/query/query_settings/query_settings_max_time_ms.js intermittently hangs until killed by Evergreen's timeout, confirmed across four sharded jscore-passthrough suites (BF-44969). Root cause confirmed via source-level tracing and local red/green reproduction — not inferred from log correlation.

      Root cause

      CurOpFailpointHelpers::waitWhileFailPointEnabled (src/mongo/db/curop_failpoint_helpers.cpp:74-79) matches candidate operations against an armed failpoint's data:

      if (data.hasField("comment") && opCtx->getComment()) {
         return opCtx->getComment()->String() == data.getStringField("comment");
      }
      const auto fpNss = NamespaceStringUtil::parseFailPointData(data, "nss"sv);
      return nss.isEmpty() || fpNss.isEmpty() || fpNss == nss;
      

      When the failpoint's data specifies a comment (to scope the block to one specific query) but the operation currently being evaluated has no comment on its OperationContext, the && short-circuits and the code falls back to namespace-only matching — so any comment-less operation on that namespace also matches, not just the intended one. This predicate is invoked from ~40 call sites across command/query execution, including CursorStage::loadBatch().

      In the failing test, jstests/hooks/run_check_orphans_are_deleted.js issues real, comment-less queries against every user collection (including the test's own) to check for orphans. When one of these lands on the collection while the test's comment-scoped failpoint happens to be armed, it gets wrongly blocked. Because FailPoint::Impl::setMode (used to disarm the failpoint) spins unconditionally until all such waiters drain their reference on the failpoint, and the wrongly-blocked orphan-check query is itself waiting for the failpoint to be disabled, this is a genuine circular-wait deadlock, confined to whichever node the race lands on.

      Fix

      File: src/mongo/db/curop_failpoint_helpers.cpp

      // Hang only the queries matching the comment field if one is specified.
      constexpr std::string_view commentFieldName = "comment"sv;
      if (auto targeted = data[commentFieldName]; !targeted.eoo()) {
         auto comment = opCtx->getComment();
         return comment && comment->woCompare(targeted) == 0;
      }
      // Hang only the queries matching the nss field if one is specified.
      constexpr std::string_view nssFieldName = "nss"sv;
      if (data.hasField(nssFieldName)) {
         return NamespaceStringUtil::parseFailPointData(data, nssFieldName) == nss;
      }
      return true;
      

      Using BSONElement::woCompare rather than forcing a string comparison also makes this robust to non-string comment values (confirmed via OperationContext::getComment()'s actual return type, boost::optional<BSONElement>), which the original .String() call was not.

            Assignee:
            Daniel Tabacaru
            Reporter:
            Daniel Tabacaru
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: