Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-12344

Add statistics tracking memory used for reconciliation and bulk load

    • Storage Engines
    • 8
    • StorEng - Defined Pipeline

      WiredTiger doesn't currently track information related to memory allocated for the process of converting in-memory content into on-disk content. This process is called reconciliation and can account for a meaningful amount of memory in some situations.

      It would be great if we could track the amount of memory associated with in-flight reconciliations in a way that doesn't interfere with performance. The information would be useful at both the connection and data handle level.

      Specifically, this probably involves tracking memory attached to WT_RECONCILE data structures as they are populated, and decrementing the memory when they are freed. Sometimes the memory is freed when reconciliation returns, but sometimes it persists past that, until an eviction is finalized.

      There is a special case of reconciliation memory associated with bulk load. Bulk load is a "special" operation that allows content to be inserted efficiently into the database. It achieves this by bypassing the cache and writing records directly onto pages. As it proceeds, leaf pages are written out, but keys necessary for the internal page structures are retained until the bulk load completes. The keys are retained in a WT_RECONCILE structure associated with the WT_CURSOR_BULK data structure.

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: