POC: Implement intermediate flushes in bulk cursor load to bound peak memory usage

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Btree, Cursors
    • None
    • Storage Engines, Storage Engines - Transactions
    • 0.001
    • None
    • None

      Current bulk cursor load accumulates all block metadata (r->multi[]) in memory until cursor close. For large tables this grows without bound — at 1 TB with 128-byte keys, peak memory exceeds 14 GB.\n\nThis POC implements a bulk_insert_flush_threshold configuration parameter that triggers periodic eviction of completed non-rightmost leaf blocks to disk during the load. The rightmost path (current insertion point) stays in memory; all other completed blocks are checkpointed and freed. This bounds peak memory to O(threshold) rather than O(table_size).\n\nStage 1 (this ticket): bounded-memory bulk load. Crash mid-load restarts from zero. Stage 2 (future work): durable intermediate checkpoints for crash resume.\n\nRelated epic: SPM-4590

            Assignee:
            Shoufu Du
            Reporter:
            Shoufu Du
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: