Initialize in-memory collection size/count metadata on recovery

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • Fully Compatible
    • Storage Execution 2026-04-27, Storage Execution 2026-05-11
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      During recovery, we want to initialize the in-memory size/count for each collection in the catalog by combining the persisted data in the config.fast_count_metadata_store with the persisted oplog
      entries after the valid-as-of timestamp. This work should enable constant time access of collection size/count by storing the size/count directly within either the CollectionImpl or the WiredTigerRecordStore as two atomics.

      To initialize the in-memory metadata, we use the following algorithm. After the last WiredTiger checkpoint has been loaded and the oplog has been replayed,
      1. Initialize a map of Collection UUID -> CollectionSizeCount
      2. Iterate over the config.fast_count_metadata_store and insert each collection's size and count into the map
      3. Read the valid-as-of timestamp
      4. Use aggregateSizeCountDeltas() to accumulate the changes to size/count since the valid-as-of timestamp
      5. Combine the aggregateSizeCountDeltas() map with the map created in step 1.
      6. Write the combined size/count to its respective collection in the collection catalog. Note that the collection catalog uses copy-on-write semantics with an immutable map storing each collection, so doing this swap in one operation would be nice.

      Also consider the space complexity of steps 1 + 4. Right now, aggregateSizeCountDeltas() is inteded to be used before readAndIncrementSizeCounts(), but we want the opposite ordering here: for all entries in the size count store, if there exists an entry in the oplog, increment it.

            Assignee:
            Cedric Sirianni
            Reporter:
            Cedric Sirianni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: