Investigate changes in SPM-4740: Atlas Infinite Continuous Internode Validation

XMLWordPrintableJSON

    • Type: Investigation
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Tools and Replicator
    • 16

      Original Downstream Change Summary

      This project will be:
      1. Changing oplog entries (Cloud, TAR)
      2. Adding log lines/fasserts (Triage & Release)
      3. Adding support to admin.system.version collection (Cloud)

      Description of Linked Ticket

      Epic Summary

      Summary

      This project will add validation that verifies that non-primary nodes are consistent with the primary node on Atlas Infinite clusters.

      Motivation

      On Atlas Infinite clusters, there is not a way to compare the data of primaries and non-primary nodes to ensure consistency. We do not have separate copies of the data that enable us to run Atlas Fleet Validation with internode validation like we do on Atlas Core clusters. We have added some runtime consistency checks on Atlas Infinite clusters as part of the replicated collection size and count project that enable us to identify some cases of data corruption, but we want to extend these checks to address the lack of internode consistency checks in Atlas Infinite clusters.

      Atlas Infinite is also where consistency risk is highest: disaggregated storage introduces new code paths including blind writes on secondaries and layered tables in WiredTiger, which increase the surface area for cross-node divergence. Some work is scheduled to address durability issues in WiredTiger, but these WiredTiger consistency checks will be probabilistic and will not address mongod durability issues. Unlike Atlas Core, Atlas Infinite cannot detect some classes of data corruption, such as in-memory bit flips of WiredTiger cache memory that do not invalidate the structure of a BSON document.

      Documentation

      Scope

      Technical Design

       

            Assignee:
            Unassigned
            Reporter:
            Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: