Current bulk cursor load accumulates all block metadata (r->multi[]) in memory until cursor close. For large tables this grows without bound — at 1 TB with 128-byte keys, peak memory exceeds 14 GB.\n\nThis POC implements a bulk_insert_flush_threshold configuration parameter that triggers periodic eviction of completed non-rightmost leaf blocks to disk during the load. The rightmost path (current insertion point) stays in memory; all other completed blocks are checkpointed and freed. This bounds peak memory to O(threshold) rather than O(table_size).\n\nStage 1 (this ticket): bounded-memory bulk load. Crash mid-load restarts from zero. Stage 2 (future work): durable intermediate checkpoints for crash resume.\n\nRelated epic: SPM-4590
POC: Implement intermediate flushes in bulk cursor load to bound peak memory usage
- Assignee:
-
Shoufu Du
- Reporter:
-
Shoufu Du
- Votes:
-
0 Vote for this issue - Watchers:
-
1 Start watching this issue
- Created:
- Updated: