Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2094

Support arbitrarily large records with unbuffered logging

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      17
    • Last comment by Customer:
      true

      Description

      The current direct write path in logging, where the read/write lock was removed, is causing issues. I'm going to redo how unbuffered log records are accounted in order to eliminate the direct write path.

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2094 Needed to rebuild the stat files.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/5bc599052dfa234415a14740d233ead2245a50fe

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2094 Needed to rebuild the stat files. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/5bc599052dfa234415a14740d233ead2245a50fe
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2094 Force unbuffered path more often.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/0f76bd0e1aac73c9904a3efc4e1193384504cdff

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2094 Force unbuffered path more often. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/0f76bd0e1aac73c9904a3efc4e1193384504cdff
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'}

          Message: WT-2094 Add unbuffered statistic.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/2d58c0a76bce34a2636a238fc90f53b8704f4e37

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'sueloverso', u'name': u'Susan LoVerso', u'email': u'sue@wiredtiger.com'} Message: WT-2094 Add unbuffered statistic. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/2d58c0a76bce34a2636a238fc90f53b8704f4e37
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2180 from wiredtiger/log-unbuffered

          WT-2094 Eliminate direct write and record unbuffered log records that
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/f883d27375b2ef8966a3ec82c63ce9a63c9ad432

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2180 from wiredtiger/log-unbuffered WT-2094 Eliminate direct write and record unbuffered log records that Branch: develop https://github.com/wiredtiger/wiredtiger/commit/f883d27375b2ef8966a3ec82c63ce9a63c9ad432
          Hide
          michael.cahill Michael Cahill added a comment -

          Sue LoVerso, I've merged the branch, thanks!

          I take it the bug yesterday was not tracking the size of unbuffered writes correctly?

          I ran the log-append.wtperf workload with varying value sizes. In the case of using 130000 value sizes, where every record will exceed our buffer, the new tree is slower by about 40% compared against an old-develop (changeset 12044d2 - before any logging changes at all). In the case of moderate sized records (24000) the two runs are within 1-2% of each other. The current log-append.wtperf using default value sizes (100 bytes) runs about 40% faster in this branch. These were run on my Mac.

          The improvements for small log writes look great! With large writes, do you understand why they are slower? We already had fixed-sized log buffers in revision 12044d2, right?

          Show
          michael.cahill Michael Cahill added a comment - Sue LoVerso , I've merged the branch, thanks! I take it the bug yesterday was not tracking the size of unbuffered writes correctly? I ran the log-append.wtperf workload with varying value sizes. In the case of using 130000 value sizes, where every record will exceed our buffer, the new tree is slower by about 40% compared against an old-develop (changeset 12044d2 - before any logging changes at all). In the case of moderate sized records (24000) the two runs are within 1-2% of each other. The current log-append.wtperf using default value sizes (100 bytes) runs about 40% faster in this branch. These were run on my Mac. The improvements for small log writes look great! With large writes, do you understand why they are slower? We already had fixed-sized log buffers in revision 12044d2, right?

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 28 weeks, 6 days ago
                Date of 1st Reply: