-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Catalog and Routing
-
ALL
-
2
-
🟥 DDL, 🟦 Shard Catalog
-
None
-
None
-
None
-
None
-
None
-
None
Important note: This can happen in suites that enable the WTWriteConflictException failpoint.
Problem
When a WriteConflict (code 112) is raised inside a sharding DDL coordinator or a resharding
state machine, nothing retries it. Two distinct outcomes follow:
- Shard crash. The resharding donor and recipient state machines end with
.then(_finishReshardingOperation).onError(LOGV2_FATAL). That handler is fatal by design,
because the shard has already committed the resharding decision locally and cannot roll back. A
single WriteConflict reaching it kills the node. - Command failure. Other DDL coordinators surface the raw WriteConflict to the caller, so
user-facing commands such as moveCollection, addShard and createUnsplittableCollection
fail outright instead of retrying internally.
Every occurrence carries the same inner error:
WriteConflict: Caused by :: Write conflict during plan execution and yielding is disabled.
Evidence
Observed across nine burn-in logs on Evergreen patch 95d82dce2855435ca2e7393d11550cad8e32c4c4,
in noPassthrough_primary_driven_index_builds on amazon-linux2023-arm64, rhel8-arm64-debug-tsan,
rhel8-64-bit-dynamic and windows.
Fatal (node crash) — 6 occurrences:
- 5551101 ReshardingRecipientService x4 — src/mongo/db/s/resharding/resharding_recipient_service.cpp:736
- 5160600 ReshardingDonorService x2 — src/mongo/db/s/resharding/resharding_donor_service.cpp:649
Command failures: - _shardsvrReshardCollection / _configsvrReshardCollection — surfaces to the user as
moveCollection failing with "Recipient shard <shard> reached an unrecoverable error" - _shardsvrCreateCollection — surfaces as createUnsplittableCollection / create failing
- _configsvrAddShard — surfaces as addShard failing, which breaks test fixture setup
- _shardsvrSetUserWriteBlockMode
Root cause
Three mechanisms combine so that no layer retries:
- writeConflictRetry deliberately does not retry when already inside an outer
WriteUnitOfWork, because the entire WUOW would have to be retried
(src/mongo/db/shard_role/lock_manager/exception_util.h:147-149). It runs the body once and lets
the exception escape, expecting an outer retry that these paths do not have. - A plan executor cannot yield inside a WriteUnitOfWork, so on conflict it calls
throwWriteConflictException() rather than retrying
(src/mongo/db/query/plan_executor_impl.cpp:637-641 and
src/mongo/db/exec/express/plan_executor_express.cpp:66-73) — hence the "yielding is disabled"
wording. - The primary-only-service retry predicate used by the donor and recipient,
kRetryabilityPredicateIncludeWriteConcernTimeout
(src/mongo/db/s/primary_only_service_helpers/with_automatic_retry.h:35-39), does not cover
WriteConflict. Error code 112 carries no error categories at all in
src/mongo/base/error_codes.yml, so status.isA<ErrorCategory::RetriableError>() is false.
By contrast WriteConflictRetryLimitExceeded (512) is categorised
[SystemOverloadedError, RetriableError] and would be retried.
The fatal handlers themselves are not the bug. The fix is to ensure WriteConflict never reaches
them, by retrying at a level that can legitimately restart the whole unit of work.Steps to reproduce
Run any resharding operation on a mongod started with the write-conflict failpoint armed:
--setParameter "failpoint.WTWriteConflictException={mode:{activationProbability:0.001}}"