Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5115

Long-running backup cursors can pin a surprising amount of metadata on disk

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: 4.2.0
    • Fix Version/s: Backlog
    • Component/s: Backup, Block manager
    • Labels:
      None

      Description

      Consider a case where there are a lot of tables with some random subset only receiving a small amount of data between checkpoints. A checkpoint modifies the file: row (due to a changing LSN and other fields, e.g: [1]), inside the WiredTiger.wt file for each table that received an update. This results in the new image of the block that row belongs to getting written out to disk. Additionally, because backup cursors stop WT from reusing data from interim checkpoints, the file growth here can be surprisingly large compared to the growth of user data that was actually inserted over the same amount of time.

      [1]

      access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=,cache_resident=false,checkpoint=(WiredTigerCheckpoint.10=(addr="018581e4341ae4908681e4424107b48781e47820e09f808080e26fc0cfc0",order=10,time=1567107756,size=8192,newest_durable_ts=0,oldest_start_ts=0,oldest_start_txn=0,newest_stop_ts=-1,newest_stop_txn=-11,write_gen=30)),checkpoint_lsn=(2,184448),checksum=uncompressed,collator=,columns=,dictionary=0,encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=2,ignore_in_memory_cache_size=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=0,log=(enabled=true),memory_page_image_max=0,memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,value_format=u,version=(major=1,minor=1)
      

        Attachments

          Activity

            People

            • Assignee:
              backlog-server-storage-engines Backlog - Storage Engines Team
              Reporter:
              daniel.gottlieb Daniel Gottlieb
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: