Knobs, flags and observability to reduce and measure checkMetadataConsistency's impact

XMLWordPrintableJSON

    • Catalog and Routing
    • Fully Compatible
    • v9.0
    • CAR Team 2026-07-20, CAR Team 2026-08-03
    • 200
    • 🟥 DDL, 🟦 Shard Catalog
    • None
    • None
    • None
    • None
    • None
    • None

      checkMetadataConsistency has gotten bigger with shards authoritative. Specifically, we are checking chunk metadata between the CSRS, shards, and the in-memory chunk map - which we did not do before. This is done under a DDL lock, preventing DDLs on the namespace. Also, this might lead to reading a lot more content from the CSRS.

      We should do at least the following:

      1. Build observability in the impact from CMC: lock timing/contention, chunk and collection count, etc
      2. Offer a flag as a means to not do a full chunk information compare. Continue to do a soft check like the count, version, etc.
      3. Offer a knob that can dial a ful check on/off - example, if less than 10k chunks, do a full check otherwise only a soft check

            Assignee:
            Jordi Olivares Provencio
            Reporter:
            Sulabh Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: