-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Catalog and Routing
-
Fully Compatible
-
CAR Team 2026-11-09
-
🟩 Routing and Topology
-
None
-
None
-
None
-
None
-
None
-
None
Objective
Modularize the build of the concrete steps used by the setFeatureCompatibilityVersion command. Give each concrete FCV step its own Bazel library, keep the registry/framework library independent of concrete implementations, and link the implementations into mongod through one aggregation target. Demonstrate that a concrete step can be linked and tested independently of all other steps and of the setFCV command implementation.
Problem and motivation
Currently, the framework lives in src/mongo/db/commands/set_feature_compatibility_version_steps, but the concrete step .cpp files are listed directly in the src/mongo/db/commands:mongod library's srcs in src/mongo/db/commands/BUILD.bazel.
Consequences:
- Adding a feature-specific step requires editing the large command BUILD file and pulling its build owners into review.
- Concrete steps inherit the command library's broad dependency environment, obscuring the libraries each step actually needs.
- A unit test cannot simply depend on one concrete step target because those targets do not exist.
- Static registration can silently disappear if a step is not linked. Build isolation needs a test that proves real dispatch, rather than a test that passes when the registry is empty.
The intended outcome is a stable build convention: adding a step means adding one library and one entry in the aggregation target, both in the steps directory.
Verified source baseline
The investigation baseline is master commit 96d713f85e8ce028711b5fae7a331d814d8fc871 from 2026-11-16. Apply the refactor to current master at implementation time; this SHA is an immutable reference, not a requirement to use an old base.
Useful source references:
- Master steps BUILD file
- Master command BUILD file
- Framework interface and Registerer
- Registry implementation
Architectural decisions
1. Retain the narrow framework target and API
Keep the existing label:
//src/mongo/db/commands/set_feature_compatibility_version_steps:set_feature_compatibility_version_steps
- Its only implementation source remains fcv_step.cpp.
- Preserve its existing framework dependencies, currently //src/mongo:base and //src/mongo/db:service_context.
- Do not add concrete steps, sharding subsystems, index-build implementations, or their dependencies to this library.
- Preserve the FCVStep, FCVStepRegistry, and FCVStepRegistry::Registerer<ActualStep> APIs and their runtime semantics.
- The existing package-default label is shorthand for this same target; keep existing framework-only consumers working.
2. One library per concrete step
Use mongo_cc_library from //bazel:mongo_src_rules.bzl. Each concrete library owns one existing step .cpp and depends on the framework plus the subsystem targets needed by that source.
At the verified master baseline, the complete inventory is:
| Target name | Source | Purpose |
|---|---|---|
| index_builds_fcv_step | index_builds_fcv_step.cpp | Reject FCV upgrade/downgrade while index builds are in progress. |
| invalid_wildcard_index_fcv_step | invalid_wildcard_index_fcv_step.cpp | Validate wildcard-index compatibility on upgrade. |
| legacy_fcv_step | legacy_fcv_step.cpp | Existing legacy feature-specific upgrade/downgrade work. |
| persistent_stats_fcv_step | persistent_stats_fcv_step.cpp | Drop persisted samples when downgrading across the persistent-statistics feature boundary. |
| set_clear_shard_uuids_fcv_step | set_clear_shard_uuids_fcv_step.cpp | Generate/clear shard UUID metadata during FCV transitions. |
Keep these source files in their current directory. The refactor changes their build ownership, not their hook implementations. If current master adds another concrete step before implementation, apply the same pattern to that step rather than leaving it in the command's srcs.
3. One source-less aggregation target
Introduce //src/mongo/db/commands/set_feature_compatibility_version_steps:fcv_step_impl as a mongo_cc_library with no sources and a deps entry for every concrete step library.
- This target represents the production link closure of all concrete steps.
- It is not the framework API target and must not become a general dependency of registry users or isolated tests.
- Put each step's subsystem dependencies on that step's library, not on the aggregation target.
- No step library or framework library may depend on fcv_step_impl or //src/mongo/db/commands:mongod. This would defeat isolation and can create dependency cycles.
The dependency graph is:
commands:mongod ----> set_feature_compatibility_version_steps (framework)
|
+-----------> fcv_step_impl
|
+----> index_builds_fcv_step --------> framework + own deps
+----> invalid_wildcard_index_fcv_step -> framework + own deps
+----> legacy_fcv_step --------------> framework + own deps
+----> persistent_stats_fcv_step ----> framework + own deps
+----> set_clear_shard_uuids_fcv_step -> framework + own deps
fcv_step_test ----------------> framework + fixture
index_builds_fcv_step_test ---> index_builds_fcv_step + framework + test support
4. Integrate through the mongod command library
Modify the mongo_cc_library(name = "mongod", ...) declaration in src/mongo/db/commands/BUILD.bazel:
- Remove all five cross-package step .cpp entries from its srcs.
- Keep set_feature_compatibility_version_command.cpp in that target.
- Add //src/mongo/db/commands/set_feature_compatibility_version_steps:fcv_step_impl to its deps.
- Retain its direct framework dependency because the command itself calls FCVStepRegistry. The two dependencies express separate responsibilities: the framework provides the API used by the command, and the aggregate links the concrete implementations.
- Preserve all other current-master command sources, dependencies, and conditional/enterprise wiring.
- A source must not remain in the command target after its concrete library is introduced: compiling it twice can duplicate static registration and symbols.
5. Preserve registration and linking behavior
Each step has a static Registerer and a ServiceContext-associated singleton. Linking its implementation must continue to run its registration constructor for each new ServiceContext; destruction must continue to unregister the step.
- Preserve step names, failpoint names, feature-flag decisions, cluster-role checks, and all hooks.
- Relative step execution order remains unspecified. Do not introduce a dependency/order API or change the FCV protocol.
- Use the repository's existing Mongo C++ build macros and linking conventions. Do not introduce a custom plugin loader, public concrete-step API, or global linker-setting changes.
- Verify static-initializer retention under both the usual build configuration and the static/archive configuration used by CI. A successfully compiled library is not proof that its registration is present in the final binary.
- After migrating the source consumers, narrow exports_files(glob(...)) to headers only. Header export is still needed by the auto-header machinery. At the baseline, the command BUILD file is the only .bazel consumer of the exported step .cpp labels.
6. Keep dependencies accurate without broadening the framework
Use the following subsystem targets as the starting point for each concrete library's dependencies. Validate the complete dependency closure with the symbol checker; the table is not an exhaustive dependency specification:
| Step | Known subsystem dependency seeds, in addition to the framework |
|---|---|
| index_builds_fcv_step | //src/mongo/db/index_builds:index_builds_coordinator |
| invalid_wildcard_index_fcv_step | //src/mongo/db/index:index_access_method; //src/mongo/db/shard_role/shard_catalog:catalog_helpers |
| legacy_fcv_step | //src/mongo/db/query/write_ops; //src/mongo/db/s:sharding_commands_d; //src/mongo/db/s:sharding_runtime_d; //src/mongo/db/timeseries:upgrade_downgrade_viewless_timeseries; //src/mongo/db/timeseries:upgrade_downgrade_viewless_timeseries_sharded_cluster; //src/mongo/otel/metrics:otel_metrics_service |
| set_clear_shard_uuids_fcv_step | //src/mongo/db/global_catalog/ddl:sharding_catalog_manager; //src/mongo/db/query/write_ops |
| persistent_stats_fcv_step | //src/mongo/db/shard_role/shard_catalog:catalog_helpers for collection iteration/drop; //src/mongo/db/query:query_knobs for the generated query feature-flag implementation. |
legacy_fcv_step.cpp also calls replicated-fast-count and other subsystem implementations. Include the defining libraries for those symbols in its dependency closure. Likewise, shard-UUID code calls sharding, transaction, and command helpers whose defining libraries must be available in its own dependency closure.
Use the symbol checker to identify missing definitions. Add the owning subsystem libraries to the affected concrete target. Do not solve errors by adding the mongod command library, every command dependency, or all concrete steps to the framework/test.
Required isolated test
Add src/mongo/db/commands/set_feature_compatibility_version_steps/index_builds_fcv_step_test.cpp to demonstrate isolated linking and registration of a real step. Use ServiceContextTest fixtures and the repository's unit-test assertion and death-test macros. Include fcv_step.h, index_builds_coordinator.h, index_builds_coordinator_mock.h, service_context_test_fixture.h, mongo/unittest/death_test.h, and mongo/unittest/unittest.h.
Its target should be:
mongo_cc_unit_test(
name = "index_builds_fcv_step_test",
srcs = ["index_builds_fcv_step_test.cpp"],
tags = ["mongo_unittest_fourth_group"],
deps = [
":index_builds_fcv_step",
":set_feature_compatibility_version_steps",
"//src/mongo/db:service_context_test_fixture",
"//src/mongo/db/index_builds:index_builds_coordinator_mock",
],
)
Add narrowly scoped test-support deps if needed by the behavior test below. The test must not link fcv_step_impl, legacy_fcv_step, or the mongod command implementation.
Test through FCVStepRegistry::get(serviceContext) rather than exposing the anonymous-namespace step class or its private hooks:
- Registration proof: add a DEATH_TEST_F which calls the upgrade validation hook without an installed IndexBuildsCoordinator, matching the death output against "indexBuildsCoordinator". It must reach the coordinator's invariant. If the step were missing from the binary, registry dispatch would be a silent no-op and this test would fail to observe the expected death.
- Happy path: install IndexBuildsCoordinatorMock on the fixture's own ServiceContext and assert both upgrade and downgrade validation hooks succeed with no builds in progress.
- Observable rejection and recovery: register an in-progress primary-driven build, assert both hooks throw BackgroundOperationInProgressForNamespace, remove the entry, and assert both hooks succeed again. This exercises real step behavior without modifying production code to add a mock seam.
The third test has an existing pattern in IndexBuildsCoordinatorTest::RegisteredPrimaryDrivenIndexBuildCountsAsInProgress. Relevant APIs and wiring:
- index_builds::primary_driven::registry(serviceContext) is declared in src/mongo/db/index_builds/primary_driven/util.h.
- Registry::add(...) and Registry::remove(buildUUID) are declared in primary_driven/registry.h.
- IndexBuildsCoordinator::set(...) connects the installed coordinator to that service context's primary-driven registry.
- noIndexBuildInProgress() is non-virtual, but its real implementation includes primary-driven registry entries in the count. Therefore no override or IndexBuildsManager seam is needed for this test.
- The primary-driven implementation target is //src/mongo/db/index_builds/primary_driven:primary_driven_index_builds. Add it as a test dependency if using its registry APIs directly.
- Keep registry state local to the test's ServiceContext and remove the entry with scope-based cleanup even when assertions fail.
- Use multiversion::GenericFCV::kLastLTS and multiversion::GenericFCV::kLatest as the upgrade endpoints, reversing them for downgrade. Follow the repository's Generic FCV reference comment convention. The hook itself does not rely on particular version numbers.
Retain fcv_step_test as a framework-only unit test. Its deps remain the narrow framework and //src/mongo/db:service_context_test_fixture. Preserve its existing assertions and registration-counter behavior.
Files and scope
Required edits/addition:
- src/mongo/db/commands/set_feature_compatibility_version_steps/BUILD.bazel: concrete libraries, aggregate, header exports, isolated test target, framework-only test wiring.
- src/mongo/db/commands/BUILD.bazel: replace direct step sources with the aggregate dependency while retaining the framework dependency.
- src/mongo/db/commands/set_feature_compatibility_version_steps/index_builds_fcv_step_test.cpp: isolated registration/behavior test.
Deliver build modularization and the tests proving isolation and registration. Keep the existing step interfaces and bodies. Broader developer documentation, hook-selection policy, FCV README reorganization, and feature migrations are outside this task's scope.
Validation plan
Framework and isolated-step unit tests
From the repository root:
bazel run +fcv_step_test bazel run +index_builds_fcv_step_test
Symbol checking and link analysis
Run the checker on the underlying _with_debug targets, which carry the check_symbol_target tag:
bazel build --config=symbol-checker \ //src/mongo/db/commands/set_feature_compatibility_version_steps:set_feature_compatibility_version_steps_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:index_builds_fcv_step_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:invalid_wildcard_index_fcv_step_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:legacy_fcv_step_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:persistent_stats_fcv_step_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:set_clear_shard_uuids_fcv_step_with_debug \ //src/mongo/db/commands/set_feature_compatibility_version_steps:fcv_step_impl_with_debug \ //src/mongo/db/commands:mongod_with_debug
.bazelrc defines this configuration with the symbol-checker aspect, output group, -skip_archive=False, and -keep_going. A failure such as "undefined but not found in self or deps" requires fixing the concrete library's dependency closure, not skipping the check.
Inspect the resolved link closure to confirm that the production command has all five steps and that the isolated test has only the index-build step. Obtain fresh static-compile and normal-link CI results; use the supported repository configurations rather than introducing per-target linker overrides.
Server build and focused integration tests
Build updated servers before running integration tests:
bazel build install-dist-test
Use the repository's python3-venv for all Python commands. The following tests provide targeted behavior coverage for this build-only change:
python3-venv/bin/python3 buildscripts/resmoke.py run --suites=multiversion \ jstests/multiVersion/genericSetFCVUsage/fcv_upgrade_downgrade_during_index_build.js \ jstests/multiVersion/genericSetFCVUsage/fcv_core/set_fcv_dry_run_mode.js \ jstests/multiVersion/genericSetFCVUsage/fcv_core/set_fcv_automatic_dryRun.js \ jstests/multiVersion/genericBinVersion/query-optimization/persistent_stats_upgrade_downgrade.js python3-venv/bin/python3 buildscripts/resmoke.py run --suites=no_passthrough \ jstests/noPassthrough/query/query-optimization/invalid_wildcard_index_fcv_upgrade.js python3-venv/bin/python3 buildscripts/resmoke.py run --suites=sharding \ jstests/sharding/set_fcv_logging.js
Provision the release binaries required by the persistent-statistics multiversion test, or run that test in an Evergreen multiversion task that already provisions them. Enable the test's required feature flags through the appropriate suite/variant. Report skipped/tag-excluded tests explicitly rather than counting them as passed.
The old shard-UUID integration test jstests/multiVersion/genericSetFCVUsage/fcv_upgrade_downgrade_manages_shard_uuids.js is excluded from the baseline multiversion suite because last-LTS is now 9.0. Do not make reviving that obsolete test a prerequisite for the build refactor. Preserve the step's existing implementation and validate its target/linking with the checks above.
Run bazel run format immediately before committing, in accordance with the repository workflow. Obtain an Evergreen patch validating the refactor against its current-master base.
Acceptance criteria
- Each of the five current concrete step .cpp files is compiled by exactly one dedicated library in the steps package.
- The framework remains independent of concrete-step implementations and their subsystem dependencies.
- The production mongod command library depends explicitly on the framework and on the source-less fcv_step_impl aggregate, and has no direct concrete-step .cpp sources.
- New steps can be added through the steps package BUILD file without another edit to the command BUILD file.
- No reverse dependency from a step/framework to the aggregate or mongod command library exists.
- The isolated index-build step test proves registration, successful validation, rejection with a build in progress, and successful validation after cleanup, without linking other concrete FCV steps.
- Framework registration, conditional registration, teardown, diagnostics, failpoints, and FCV behavior remain unchanged.
- Symbol-checker, server build, focused integration coverage, and fresh normal/static CI checks pass; any environment blocker or excluded test is reported with a concrete explanation.
- The build refactor and its tests form a complete, independently landable change.