Limit memory utilization of WT cursors in index build

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Execution
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Currently, index builds utilize a WiredTiger bulk cursor (link), however as evidenced in WT-15594, when using the bulk load cursor we incur memory usage that scales with the number of leaf pages used. For very large indexes, it is possible that this can cause a server OOM crash, after which the index build would need to be restarted from the beginning.

      As discussed in that ticket and in this document (internal link), we should instead capture an error from WiredTiger when we are approaching some memory limit when using the bulk cursor, and instead switch to using a non-bulk cursor, in which memory usage respects the limits of the WiredTiger cache.

            Assignee:
            Unassigned
            Reporter:
            Alex Sarkesian
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: