Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2007

Consider how much memory is consumed by log buffers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      9
    • Last comment by Customer:
      true

      Description

      Each log slot has a buffer associated with it. That buffer is grown dynamically until it is large enough to hold the consolidated records.

      We recently bumped the number of slots, which may alter how much memory log buffers consume.

      We should consider limiting how much memory the log system can use (maybe a log cache size?). Or alternatively limiting the size log buffers can grow to - there is almost certainly a limit to how much memory it makes sense to copy and consolidate into a buffer rather than just writing it directly.

        Issue Links

          Activity

          Hide
          sue.loverso Sue LoVerso added a comment -

          I think this is a good idea. I'll implement a maximum buffer size. Anything too large takes direct write. There are a number of paths we could take:
          1. Keep current code, add in a maximum and use current buffer grow code until we hit the maximum buffer size. This is the smallest code change.
          2. Just pick a maximum size overall and allocate it to each buffer once. Simplifies code and gets rid of buffer growth code and functions.
          3. Add a config knob to allow the user to modify how much memory to allocate to the log buffers. Then implement #2 with that value.

          If we add a config knob, we ought to consider a knob to control how to allocate the slots. Instead of a static number of slots in the pool the user may want fewer/larger or many/smaller depending on how large they expect their transactions to be.

          Show
          sue.loverso Sue LoVerso added a comment - I think this is a good idea. I'll implement a maximum buffer size. Anything too large takes direct write. There are a number of paths we could take: 1. Keep current code, add in a maximum and use current buffer grow code until we hit the maximum buffer size. This is the smallest code change. 2. Just pick a maximum size overall and allocate it to each buffer once. Simplifies code and gets rid of buffer growth code and functions. 3. Add a config knob to allow the user to modify how much memory to allocate to the log buffers. Then implement #2 with that value. If we add a config knob, we ought to consider a knob to control how to allocate the slots. Instead of a static number of slots in the pool the user may want fewer/larger or many/smaller depending on how large they expect their transactions to be.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Statically allocate log slot buffers to a maximum size.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/f148e05aec120ef96db54f67a08a0b760f7512f2

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Statically allocate log slot buffers to a maximum size. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/f148e05aec120ef96db54f67a08a0b760f7512f2
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Cap slot buffer size to max log file size.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/7dac9988d032bb774a85a1a808f1b1cbb5b3bbdf

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Cap slot buffer size to max log file size. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/7dac9988d032bb774a85a1a808f1b1cbb5b3bbdf
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Fix constant.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/6a40057fe6e772ff19303547d0c4499f0ed33f72

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Fix constant. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/6a40057fe6e772ff19303547d0c4499f0ed33f72
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Reinstate attempting a few times to find a slot.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/63a9a6fd9b49c4a2a88986ad9387d361b42d8847

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Reinstate attempting a few times to find a slot. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/63a9a6fd9b49c4a2a88986ad9387d361b42d8847
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Remove unused ret.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/9da3ce5c7f7feca7bba90f260f125ad403d8fdd2

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Remove unused ret. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/9da3ce5c7f7feca7bba90f260f125ad403d8fdd2
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'}

          Message: Merge pull request #2062 from wiredtiger/log-buf-static-max

          WT-2007 Statically allocate log slot buffers to a maximum size.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/e1d8bc7fa41efe1d4765cca2b9857907802fcf1b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'} Message: Merge pull request #2062 from wiredtiger/log-buf-static-max WT-2007 Statically allocate log slot buffers to a maximum size. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e1d8bc7fa41efe1d4765cca2b9857907802fcf1b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2007 Fix gcc warnings.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/27c18d300f38e16e09cfb5c826c7844120e0b11f

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2007 Fix gcc warnings. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/27c18d300f38e16e09cfb5c826c7844120e0b11f
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'}

          Message: WT-2007 Statically allocate log slot buffers to a maximum size.

          Merge pull request #2062 from wiredtiger/log-buf-static-max

          (cherry picked from commit e1d8bc7fa41efe1d4765cca2b9857907802fcf1b)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/fad2eacb13ebeb0dfd3a001ca45d82eff83a9af3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'sueloverso', u'email': u'sue@mongodb.com'} Message: WT-2007 Statically allocate log slot buffers to a maximum size. Merge pull request #2062 from wiredtiger/log-buf-static-max (cherry picked from commit e1d8bc7fa41efe1d4765cca2b9857907802fcf1b) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/fad2eacb13ebeb0dfd3a001ca45d82eff83a9af3

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 47 weeks, 2 days ago
                Date of 1st Reply: