Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2029

Improve scalability of statistics updates

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: WT2.6.1
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      5
    • Last comment by Customer:
      true

      Description

      We have measured the overhead of statistics updates in some workloads at 10% or more. However, MongoDB always runs with statistics enabled.

      Investigate reducing the overhead of statistics by splitting updates to separate locations and aggregrating when statistics are queried.

        Issue Links

          Activity

          Hide
          fedorova Alexandra (Sasha) Fedorova added a comment -

          The chart shows performance in microseconds/op for a LevelDB readseq benchmark on the WiredTiger library, on a machine with 24 cores. The purpose is to demonstrate improved performance with a new implementation of statistics counters.

          The blue curve shows an old implementation of the statistics counters, where the counters are globally shared by all threads. When statistics are turned on, this benchmark does not scale: the operations take longer to complete as we add more threads due to hardware contention for the shared statistics variables.

          The green curve shows the same experiment on the same branch, but with statistics disabled. We can see that we get nearly perfect linear scaling when statistics are disabled.

          The yellow curve shows the performance of the same benchmark, but using the scalable implementation of statistics counters (it is a variant of per-thread counters). Here, even if we enable statistics we get nearly perfect linear scalability.

          Show
          fedorova Alexandra (Sasha) Fedorova added a comment - The chart shows performance in microseconds/op for a LevelDB readseq benchmark on the WiredTiger library, on a machine with 24 cores. The purpose is to demonstrate improved performance with a new implementation of statistics counters. The blue curve shows an old implementation of the statistics counters, where the counters are globally shared by all threads. When statistics are turned on, this benchmark does not scale: the operations take longer to complete as we add more threads due to hardware contention for the shared statistics variables. The green curve shows the same experiment on the same branch, but with statistics disabled. We can see that we get nearly perfect linear scaling when statistics are disabled. The yellow curve shows the performance of the same benchmark, but using the scalable implementation of statistics counters (it is a variant of per-thread counters). Here, even if we enable statistics we get nearly perfect linear scalability.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: WT-2029, improve scalability of statistics

          Instead of using per-thread arrays of counters (see #2102), use arrays
          of statistics structures. I started with the scalable_stats branch, so
          this incorporates most of the commits on that branch, but the diff is
          against the develop branch to make it easier to review.

          Specifically, this includes a9675d0:
          There are only 3 statistics we maintain atomically: the count
          of open cursors and the split-stash bytes and object counts.
          Move them to WT_CONNECTION_IMPL fields, and copy them into the
          statistics on demand, this allows us to remove support for
          atomically maintaining statistics values.

          And 6bda287:
          Get rid of the WT_CONN_

          {STAT,STAT_GET,STAT_SET}

          macros, instead
          maintain a few additional fields in the WT_CONNECTION_IMPL
          structure (checkpoint min/max, recent and total time).

          And 7590930:
          Use a prime number of buckets rather than assuming a good hash
          (Reference Sedgewick, Algorithms in C, "Hash Functions").

          And 63a2082:
          I'm seeing a roughly 20% advantage of using a loop to clear the
          stats values, vs. a memset to clear the whole padded chunk, and
          I'd rather err on the side of writing fewer cache lines, anyway.

          Additionally:

          Now we have arrays of statistics structures, separate out each statistic
          value's description into a separate array, a struct of strings. We could
          access it directly, and maybe should, but I'm using a function all at
          the moment.

          Now the statistic's descriptions are separate from the statistics values,
          we no longer need macros to access the fields, in general we can simply
          use the stat structure reference and the field name and set the int64_t
          directly. This includes removal of the WT_STATS typedef.

          Change the LSM code to not maintain a local data-source statistics
          structure in each LSM-tree structure – that allows us to remove all of
          the macros to update a single stats structure, and I don't think it
          complicates things that much. The changes to the LSM statistics code
          should be closely reviewed.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/87bb029591a9fc8a4dad84549502c91416c4d139

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2029 , improve scalability of statistics Instead of using per-thread arrays of counters (see #2102), use arrays of statistics structures. I started with the scalable_stats branch, so this incorporates most of the commits on that branch, but the diff is against the develop branch to make it easier to review. Specifically, this includes a9675d0: There are only 3 statistics we maintain atomically: the count of open cursors and the split-stash bytes and object counts. Move them to WT_CONNECTION_IMPL fields, and copy them into the statistics on demand, this allows us to remove support for atomically maintaining statistics values. And 6bda287: Get rid of the WT_CONN_ {STAT,STAT_GET,STAT_SET} macros, instead maintain a few additional fields in the WT_CONNECTION_IMPL structure (checkpoint min/max, recent and total time). And 7590930: Use a prime number of buckets rather than assuming a good hash (Reference Sedgewick, Algorithms in C, "Hash Functions"). And 63a2082: I'm seeing a roughly 20% advantage of using a loop to clear the stats values, vs. a memset to clear the whole padded chunk, and I'd rather err on the side of writing fewer cache lines, anyway. Additionally: Now we have arrays of statistics structures, separate out each statistic value's description into a separate array, a struct of strings. We could access it directly, and maybe should, but I'm using a function all at the moment. Now the statistic's descriptions are separate from the statistics values, we no longer need macros to access the fields, in general we can simply use the stat structure reference and the field name and set the int64_t directly. This includes removal of the WT_STATS typedef. Change the LSM code to not maintain a local data-source statistics structure in each LSM-tree structure – that allows us to remove all of the macros to update a single stats structure, and I don't think it complicates things that much. The changes to the LSM statistics code should be closely reviewed. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/87bb029591a9fc8a4dad84549502c91416c4d139
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: Merge pull request #2122 from wiredtiger/scalable_stats_keith

          WT-2029, improve scalability of statistics
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/6b84722633b4d5340ed4c53b9b5251e9e191581c

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: Merge pull request #2122 from wiredtiger/scalable_stats_keith WT-2029 , improve scalability of statistics Branch: develop https://github.com/wiredtiger/wiredtiger/commit/6b84722633b4d5340ed4c53b9b5251e9e191581c
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: WT-2029, minor cleanups.

          Initialize the data-handle statistics when creating the data-handle,
          rather than when configuring the WT_BTREE handle.

          Move handle statistics initialization into the auto-generated code,
          it's cleaner if less outside code knows about WT_COUNTER_SLOTS.

          Minor whitespace cleanup.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/9265b1a4df7602017b789de52dc842e9e9589328

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2029 , minor cleanups. Initialize the data-handle statistics when creating the data-handle, rather than when configuring the WT_BTREE handle. Move handle statistics initialization into the auto-generated code, it's cleaner if less outside code knows about WT_COUNTER_SLOTS. Minor whitespace cleanup. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/9265b1a4df7602017b789de52dc842e9e9589328
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2125 from wiredtiger/stat-init-cleanup

          WT-2029, minor cleanups.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/f752f07f8c6c9ef3b5f030e0999a79d32b032c43

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2125 from wiredtiger/stat-init-cleanup WT-2029 , minor cleanups. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/f752f07f8c6c9ef3b5f030e0999a79d32b032c43

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 31 weeks, 1 day ago
                Date of 1st Reply: