ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Engines - Foundations
    • 146.311
    • None
    • None

      Problem

      WT_LAYERED_TABLE_MANAGER.entries (src/include/connection.h:130) is indexed directly by the namespaced ingest_id and sized 2 x ingest_id x 8B (src/conn/conn_layered_table_manager.c:83-86).

      File IDs carry a 3-bit namespace tag in the low bits (WT_BTREE_ID_NAMESPACED, src/include/btree.h:106-107), and ingest btrees are always tagged LOCAL (fixed tag, never SHARED - asserted at src/include/btree_inline.h:332-333). So 7 of every 8 array slots can never be occupied by an ingest entry. Combined with the realloc's blind x2 headroom, effective cost is 128 bytes of permanent RSS per raw file-ID allocated on this connection, ever - not per live layered table.

      Two clusters measured: raw counters of 1.55M and 2.02M (accumulated over roughly a week of ordinary create/drop churn - no restart or inherited-counter jump required) produced 189 MiB and 247 MiB arrays holding only ~8.6k live entries - occupancy 0.03%. The array never shrinks (drop only NULLs the slot, src/conn/conn_layered_table_manager.c:125) and IDs are never reused, so this is monotonic for the life of the connection/cluster.

      Fix (graduated, cheapest first)

      1. Index by WT_BTREE_ID_UNNAMESPACED(ingest_id) instead of the raw namespaced ID - removes the x8 waste, one-line change. 189 -> 24 MiB on the measured cluster.
      2. Drop the blind x2 realloc headroom. 24 -> 12 MiB.
      3. Replace the array with the existing WT_HASH_MAP (src/include/hash_map.h, src/support/hash_map.c), keyed on ingest_id, sized by live table count - immune to ID-space growth entirely (~138 KB regardless of churn). Needs one addition (__wt_hash_map_remove - not currently present) and a design call for the GC-pruning walk (src/conn/conn_layered_ingest.c:1244), which currently iterates the raw array and would need map iteration support or a side list of live entries.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: