Description generated via Claude after investigating a failing Automation test. It's not immediately clear to me if this is a known/expected behavior or an inadvertent version upgrade issue, so filing a ticket out of precaution. From an Automation standpoint, this seems it will self-resolve when we test with the version pair (rc1017, rc1018)

Affected version: mongodb-mongo-dsc-master 9.0.0-rc1017 (9.0.0-rc1016 unaffected)

Summary

rc1017 creates a new unique index {uuid: 1} (uuid_1) on config.shards from ShardingCatalogManager's step-up path. When the CSRS was initialized by an earlier binary, config.shards is already populated, so the index must be built over existing data. That build throws, and the abort path fasserts:

  I ELECTION 21450   Election succeeded, assuming primary role {"term": 13}
  I SHARDING 5173300 Creating index on sharding collection with existing data
                     {"namespace": "config.shards", "uuid": {"$uuid": "4e7a778c-..."},
                      "index": {"v": 2, "unique": true, "key": {"uuid": 1}, "name": "uuid_1"}}
  I INDEX 20384      Index build: starting {"namespace": "config.shards", "method": "foreground"}
  I INDEX 20391      Index build: collection scan done {"totalRecords": 2, "readSource": "kNoTimestamp"}
  E WT 22435         __wt_txn_timestamp_usage_check:652:file:_mdb_catalog.wt: unexpected timestamp
                     usage: no timestamp provided for an update to a table configured to always use
                     timestamps once they are first used
  F ASSERT 23083     Invariant failure {"expr": "wtRet",
                     "error": "BadValue: 22: Invalid argument - session ID: 27, txn ID: 71, ...
                               name: WT_SESSION.commit_transaction",
                     "location": "src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:469:35:
                                  void mongo::WiredTigerRecoveryUnit::_txnClose(bool)"}
  F ASSERT 23084     ***aborting after invariant() failure

  Elapsed time from assuming primary role to abort is ~17 ms.

Stack

mongo::error_details::invariantOKFailed(...)  mongo::WiredTigerRecoveryUnit::doCommitUnitOfWork(boost::optional<mongo::Timestamp>)
mongo::WriteUnitOfWork::commit()
mongo::MultiIndexBlock::abortIndexBuild(OperationContext*, CollectionWriter&, std::function<void()>)
mongo::IndexBuildsManager::abortIndexBuild(OperationContext*, CollectionWriter&, UUID const&, std::function<void()>) mongo::ScopeGuard<mongo::IndexBuildsCoordinator::_createIndex(...)::$_2>::~ScopeGuard()

Two distinct problems

  1. The fassert (primary defect). MultiIndexBlock::abortIndexBuild commits a WriteUnitOfWork that updates mdb_catalog without a timestamp, tripping WT's timestamp-usage check and aborting the process. A failed index build should surface as an error, not kill the node. This is latent for _any failing config-index build on step-up, independent of the uuid_1 change.

  2. Upgrade incompatibility. Building unique config.shards.uuid_1 over shard documents written by a pre-uuid binary fails. Hypothesis (not isolated): the pre-existing config.shards docs have no uuid field, so all of them index as null and collide in the unique index — with two shards that's an immediate E11000. The failure reason is masked because the abort path crashes before it can be logged.

Impact  

Any deployment whose config DB predates the uuid_1 index crashes its config servers on the first step-up after upgrading to rc1017, and does so repeatedly: whichever node is elected dies instantly, the pre-upgrade node reclaims the primary after its replSetStepDown freeze expires, and the cycle repeats. In our CI run this produced 160 consecutive crash-loop cycles across two

config nodes over ~46 minutes (82 on cs2, 78 on cs3) with no forward progress — the rolling upgrade can never drain the last old-version primary, so it never completes.

Reproduction

Observed in Cloud Automation CI (ShFullSuite.TestUpgradeDuringShardDrain, disagg variant):

  1. Start a 3-node CSRS plus two 3-node shards on 9.0.0-rc1016; add both shards; shard a collection.

  2. Rolling-upgrade all binaries to 9.0.0-rc1017.

  3. The first rc1017 config node to win an election aborts on the invariant above.

The originating test additionally has a stalled removeShard drain in progress (balancer disabled, 5 chunks remaining on the draining shard) — we do not believe that is required for the crash, only that config.shards be non-empty and lack uuid_1. A ≥2-shard cluster upgraded rc1016 → rc1017 should be sufficient.

Evidence that this is new in rc1017

Within the same CI task:

  - cs1 stayed on rc1016 (it could never hand off the primary), stepped up 21 times, never attempted the uuid_1 build, never crashed. On rc1016, config.shards carries only id and host_1.

  - Other clusters in the task were initialized natively on rc1017: they create uuid_1 at init while config.shards is empty, log no Creating index on sharding collection with existing data, step up, and do not crash.

  - The only change between the last passing and first failing CI revision is the test-version bump 9.0.0-rc1016 → 9.0.0-rc1017 (mms-automation commit 7027be669f, CLOUDP-429118) — no server-facing agent code changed.

Artifacts

Task: cloud_automation_master_al2023_amd64_atlas_disagg_asc_TestGroup35_e40d2a2b5e0d64a3d6f2d334cdafe337098a66d3_26_08_04_20_31_19, execution 2.

  Crashing mongod logs in artifacts.tgz under go_planner/tmp/logs/:

  - ShFullSuite_TestUpgradeDuringShardDrain_run9002_4.* (cs2, rc1017 — 82 crashes)

  - ShFullSuite_TestUpgradeDuringShardDrain_run9003_4.* (cs3, rc1017 — 78 crashes)

  - ShFullSuite_TestUpgradeDuringShardDrain_run9001_4 (cs1, rc1016 — control, no crash)

  Representative single cycle: ..._run9002_4.2026-08-05T05-00-49 (start 05:00:12, election 05:00:24.401, abort 05:00:24.418).