Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2975

Reduce the memory allocated for statistics

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3
    • Labels:
      None
    • # Replies:
      19
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-11-21

      Description

      The WiredTiger statistics tracking structure uses 64-bit values, and an array of 23 entries per value to allow for efficiently updating statistics across many threads.

      The consequence of that is that each statistic requires 184 bytes in each dhandle. The dhandle currently maintains approximately 100 statistics, leading to 18k of memory required for each dhandle. Some applications open over 100 thousand handles - which can mean nearly 2GB of memory allocated just for dhandle statistics structures.

      We should consider mechanisms for reducing the memory footprint of our statistics. A possible path forward would be to only keep arrays of values for statistics that are updated across multiple threads in parallel, that's not ideal, since it means identifying that set of statistics (thus potentially adding in performance bottle necks) and will make the statistics tracking macros more complex, but it could save a significant amount of space.

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2975 Fix a leak of statistics data when reopening handles. (#3114)
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/c726a283cb77742807412efff2b2e0ed2f7025a9

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2975 Fix a leak of statistics data when reopening handles. (#3114) Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/c726a283cb77742807412efff2b2e0ed2f7025a9
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2975 Only allocate space for statistics when we open a data source (#3107)
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/c403455d8bb87d3c5a99a327a7aa75bdbf80cd4a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2975 Only allocate space for statistics when we open a data source (#3107) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/c403455d8bb87d3c5a99a327a7aa75bdbf80cd4a
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2975 Fix a leak of statistics data when reopening handles. (#3114)
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/c726a283cb77742807412efff2b2e0ed2f7025a9

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2975 Fix a leak of statistics data when reopening handles. (#3114) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/c726a283cb77742807412efff2b2e0ed2f7025a9
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

          ref: ef9a7983ea..74430da40c
          for: 3.4.0

          SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
          WT-2415 Add option for join cursors to return false positives
          WT-2945 Occasional hang running reconfigure fuzz test
          WT-2947 test suite populate functions need refactoring
          WT-2953 Test format failure - stuck cache with 1/42MB dirty
          WT-2961 Add a version drop-down to the web version of the docs
          WT-2964 WTPERF parallel-pop-btree hung in populate phase
          WT-2968 backup cursor opens every file
          WT-2971 Add documentation for raw compression to the WT docs
          WT-2975 Reduce the memory allocated for statistics
          WT-2991 Coverity issue 1365198
          WT-2998 Add error messages to error returns that might be confusing.
          WT-2999 Memory leak in join WT_CURSOR with custom extractor
          WT-3001 WT_EXTENSION_API references are named inconsistently.
          WT-3003 log generation should not auto generate the record and op types
          WT-3005 Add top-level .gitignore file.
          WT-3007 Remove duplicated word in comment
          WT-3008 Run wtperf stress testing against all maintained branches
          WT-3009 Test format hung/aborted with small cache (sub 10mB)
          Branch: master
          https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4 ref: ef9a7983ea..74430da40c for: 3.4.0 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw compression to the WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: master https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

          ref: b11ed312ce..2e2d5fe23b
          for: 3.2.11

          SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
          WT-2415 Add option for join cursors to return false positives from a Bloom filter
          WT-2945 Occasional hang running reconfigure fuzz test
          WT-2947 test suite populate functions need refactoring
          WT-2953 Test format failure - stuck cache with 1/42MB dirty
          WT-2961 Add a version drop-down to the web version of the docs
          WT-2964 WTPERF parallel-pop-btree hung in populate phase
          WT-2968 backup cursor opens every file
          WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
          WT-2975 Reduce the memory allocated for statistics
          WT-2991 Coverity issue 1365198
          WT-2998 Add error messages to error returns that might be confusing.
          WT-2999 Memory leak in join WT_CURSOR with custom extractor
          WT-3001 WT_EXTENSION_API references are named inconsistently.
          WT-3003 log generation should not auto generate the record and op types
          WT-3005 Add top-level .gitignore file.
          WT-3007 Remove duplicated word in comment
          WT-3008 Run wtperf stress testing against all maintained branches
          WT-3009 Test format hung/aborted with small cache (sub 10mB)
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2 ref: b11ed312ce..2e2d5fe23b for: 3.2.11 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives from a Bloom filter WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: v3.2 https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                28 weeks, 4 days ago
                Date of 1st Reply:

                  Agile