ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • 2
    • 🟥 DDL, 🟦 Shard Catalog
    • None
    • None
    • None
    • None
    • None
    • None

      Important note: This can happen in suites that enable the WTWriteConflictException failpoint.

      Problem

      When a WriteConflict (code 112) is raised inside a sharding DDL coordinator or a resharding
      state machine, nothing retries it. Two distinct outcomes follow:

      • Shard crash. The resharding donor and recipient state machines end with
        .then(_finishReshardingOperation).onError(LOGV2_FATAL). That handler is fatal by design,
        because the shard has already committed the resharding decision locally and cannot roll back. A
        single WriteConflict reaching it kills the node.
      • Command failure. Other DDL coordinators surface the raw WriteConflict to the caller, so
        user-facing commands such as moveCollection, addShard and createUnsplittableCollection
        fail outright instead of retrying internally.
        Every occurrence carries the same inner error:
        WriteConflict: Caused by :: Write conflict during plan execution and yielding is disabled.

      Evidence

      Observed across nine burn-in logs on Evergreen patch 95d82dce2855435ca2e7393d11550cad8e32c4c4,
      in noPassthrough_primary_driven_index_builds on amazon-linux2023-arm64, rhel8-arm64-debug-tsan,
      rhel8-64-bit-dynamic and windows.
      Fatal (node crash) — 6 occurrences:

      • 5551101 ReshardingRecipientService x4 — src/mongo/db/s/resharding/resharding_recipient_service.cpp:736
      • 5160600 ReshardingDonorService x2 — src/mongo/db/s/resharding/resharding_donor_service.cpp:649
        Command failures:
      • _shardsvrReshardCollection / _configsvrReshardCollection — surfaces to the user as
        moveCollection failing with "Recipient shard <shard> reached an unrecoverable error"
      • _shardsvrCreateCollection — surfaces as createUnsplittableCollection / create failing
      • _configsvrAddShard — surfaces as addShard failing, which breaks test fixture setup
      • _shardsvrSetUserWriteBlockMode

      Root cause

      Three mechanisms combine so that no layer retries:

      1. writeConflictRetry deliberately does not retry when already inside an outer
        WriteUnitOfWork, because the entire WUOW would have to be retried
        (src/mongo/db/shard_role/lock_manager/exception_util.h:147-149). It runs the body once and lets
        the exception escape, expecting an outer retry that these paths do not have.
      2. A plan executor cannot yield inside a WriteUnitOfWork, so on conflict it calls
        throwWriteConflictException() rather than retrying
        (src/mongo/db/query/plan_executor_impl.cpp:637-641 and
        src/mongo/db/exec/express/plan_executor_express.cpp:66-73) — hence the "yielding is disabled"
        wording.
      3. The primary-only-service retry predicate used by the donor and recipient,
        kRetryabilityPredicateIncludeWriteConcernTimeout
        (src/mongo/db/s/primary_only_service_helpers/with_automatic_retry.h:35-39), does not cover
        WriteConflict. Error code 112 carries no error categories at all in
        src/mongo/base/error_codes.yml, so status.isA<ErrorCategory::RetriableError>() is false.
        By contrast WriteConflictRetryLimitExceeded (512) is categorised
        [SystemOverloadedError, RetriableError] and would be retried.
        The fatal handlers themselves are not the bug. The fix is to ensure WriteConflict never reaches
        them, by retrying at a level that can legitimately restart the whole unit of work.

        Steps to reproduce

        Run any resharding operation on a mongod started with the write-conflict failpoint armed:

        --setParameter "failpoint.WTWriteConflictException={mode:{activationProbability:0.001}}"
        

            Assignee:
            Unassigned
            Reporter:
            Igor Praznik
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: