-
Type:
Investigation
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Tools and Replicator
-
16
This project will be:
1. Changing oplog entries (Cloud, TAR)
2. Adding log lines/fasserts (Triage & Release)
3. Adding support to admin.system.version collection (Cloud)
Description of Linked Ticket
Summary
This project will add validation that verifies that non-primary nodes are consistent with the primary node on Atlas Infinite clusters.
Motivation
On Atlas Infinite clusters, there is not a way to compare the data of primaries and non-primary nodes to ensure consistency. We do not have separate copies of the data that enable us to run Atlas Fleet Validation with internode validation like we do on Atlas Core clusters. We have added some runtime consistency checks on Atlas Infinite clusters as part of the replicated collection size and count project that enable us to identify some cases of data corruption, but we want to extend these checks to address the lack of internode consistency checks in Atlas Infinite clusters.
Atlas Infinite is also where consistency risk is highest: disaggregated storage introduces new code paths including blind writes on secondaries and layered tables in WiredTiger, which increase the surface area for cross-node divergence. Some work is scheduled to address durability issues in WiredTiger, but these WiredTiger consistency checks will be probabilistic and will not address mongod durability issues. Unlike Atlas Core, Atlas Infinite cannot detect some classes of data corruption, such as in-memory bit flips of WiredTiger cache memory that do not invalidate the structure of a BSON document.
Documentation