Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16665

WiredTiger b-tree uses too much space in journal directory

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major - P3
    • Resolution: Done
    • 2.8.0-rc3
    • None
    • Storage, WiredTiger
    • Storage Execution
    • Fully Compatible

    Description

      I am running the iibench for MongoDB on two hosts. Both use the WT b-tree, one with snappy and the other with zlib compression. The test uses 10 threads to load 200M documents per thread. Immediately after the load is done the zlib test uses ~500G versus ~330G for snappy. With some idle time the zlib disk usage drops to ~160G. At first I thought the problem was only for zlib, but watching the size of data/journal during tests shows problems for snappy and zlib.

      My test host has 40 hyperthread cores, 144G RAM and fast (PCIe) flash

      Can WiredTiger delete log files sooner?

      I have a test in progress and it is about 30% done. The problem for the zlib configuration is too much space used in the journal directory. Note that "data" is the root for database files:

      du -hs data data/journal; ls data/journal | wc -l
      		 
      163G    data
      		 
      131G    data/journal
      		 
      1337

      And this is from the snappy test at about 20% done:

      du -hs data data/journal; ls data/journal | wc -l
      		 
      69G     data
      		 
      23G     data/journal
      		 
      231

      And this is from later in the snappy test:

      du -hs data data/journal; ls data/journal | wc -l
      		 
      120G    data
      		 
      47G     data/journal
      		 
      475


      ls -lh data/journal/
      		 
      total 25G
      		 
      -rw-r--r-- 1 root root 100M Dec 24 07:58 WiredTigerLog.0000001012
      		 
      -rw-r--r-- 1 root root 100M Dec 24 07:58 WiredTigerLog.0000001013
      		 
      -rw-r--r-- 1 root root 100M Dec 24 07:58 WiredTigerLog.0000001014
      		 
      <snip>

      And later in the zlib test:

      du -hs data data/journal; ls data/journal | wc -l
      		 
      231G    data
      		 
      182G    data/journal
      		 
      1862

      Attachments

        1. es.022553
          13 kB
        2. metrics.2016-05-31T03-47-19Z-00000.gz
          2.34 MB

        Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. SERVER-16736 support more than 1 checkpoint thread for WiredTiger

          • Major - P3 - Major loss of function.
          • Backlog

          New Feature - A new feature of the product, which has yet to be developed. SERVER-16737 support eviction_dirty_trigger for WiredTiger

          • Major - P3 - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. WT-2764 Optimize checkpoints to reduce throughput disruption

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. WT-2389 Spread Out I/O Rather Than Spiking During Checkpoints

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-16575 intermittent slow inserts with WiredTiger b-tree

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              backlog-server-execution Backlog - Storage Execution Team
              mdcallag Mark Callaghan
              Votes:
              2 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: