Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2950

Inserting multi-megabyte values can cause large in-memory pages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc1
    • Labels:
    • # Replies:
      8
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-10-31

      Description

      Constructing a workload that:

      • Inserts 16MB values from 10 threads in parallel
      • Configures a memory_page_max of 5MB
      • Does not do checkpoints
      • Does not use explicit transactions

      Can lead to in-memory pages growing to over 1GB. That is bad for a number of reasons, including:

      • Reconciliation will need to allocate a large amount of memory when writing the page out.
      • Reconciliation will be slow, leading to application stalls.
      1. wt2950_single_thread.png
        29 kB
      2. wt2950.png
        44 kB

        Issue Links

          Activity

          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          A wtperf workload that can reproduce the behavior is:

          conn_config="cache_size=7GB,eviction=(threads_min=4,threads_max=4),log=(enabled=false),transaction_sync=(enabled=false),checkpoint_sync=false,checkpoint=(wait=3000),statistics=(fast,cache_walk),statistics_log=(wait=10,sources=[file:],json=true)"
          table_config="allocation_size=4k,memory_page_max=5MB,prefix_compression=false,split_pct=75,leaf_page_max=32k,internal_page_max=16k,type=file,leaf_value_max=32M"
          table_count=1
          compression=snappy
          icount=100
          populate_threads=1
          reopen_connection=true
          report_interval=1
          run_time=120
          threads=((count=10,inserts=1))
          value_sz=16000000
          sample_interval=5
          sample_rate=1
          

          The resulting maximum page size seen at eviction is shown in the following graph:

          Show
          alexander.gorrod Alexander Gorrod added a comment - A wtperf workload that can reproduce the behavior is: conn_config="cache_size=7GB,eviction=(threads_min=4,threads_max=4),log=(enabled=false),transaction_sync=(enabled=false),checkpoint_sync=false,checkpoint=(wait=3000),statistics=(fast,cache_walk),statistics_log=(wait=10,sources=[file:],json=true)" table_config="allocation_size=4k,memory_page_max=5MB,prefix_compression=false,split_pct=75,leaf_page_max=32k,internal_page_max=16k,type=file,leaf_value_max=32M" table_count=1 compression=snappy icount=100 populate_threads=1 reopen_connection=true report_interval=1 run_time=120 threads=((count=10,inserts=1)) value_sz=16000000 sample_interval=5 sample_rate=1 The resulting maximum page size seen at eviction is shown in the following graph:
          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          The same workload running single threaded has more reasonable behavior, allowing pages to grow to a maximum of 32MB:

          Show
          alexander.gorrod Alexander Gorrod added a comment - The same workload running single threaded has more reasonable behavior, allowing pages to grow to a maximum of 32MB:
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081)
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081) Branch: develop https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633
          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          Thanks Michael Cahill I've merged the change. I'm a little bit concerned about performance consequences, but the benefit outweighs that at the moment. If we see workloads where there is a degradation we can use them to fine-tune how aggressive we are at releasing memory.

          Show
          alexander.gorrod Alexander Gorrod added a comment - Thanks Michael Cahill I've merged the change. I'm a little bit concerned about performance consequences, but the benefit outweighs that at the moment. If we see workloads where there is a degradation we can use them to fine-tune how aggressive we are at releasing memory.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081)
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081) Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: 6a31c2118cce88c68281eda2ca9ab1df915a2773 from branch mongodb-3.4

          ref: fc0e7abe82..6a31c2118c
          for: 3.4.0

          WT-2831 Skip creating a checkpoint if there have been no changes
          WT-2858 rename wtperf's CONFIG structure
          WT-2895 Reduce the runtime of make check testing with disable long
          WT-2907 Bug in Java ConcurrentCloseTest case
          WT-2917 split wtperf's configuration into per-database and per-run parts
          WT-2920 New eviction statistics
          WT-2931 Configure default in-memory dirty cache usage lower
          WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration
          WT-2933 Fix a race between named snapshots and checkpoints
          WT-2937 test_inmem01 aborts due to stuck cache
          WT-2938 Assembly files should end in .sx, not .S
          WT-2941 Improve test/format to use faster key-generation functions
          WT-2942 verbose strings don't need newline
          WT-2946 dist/s_docs incompatible with OS X Xcode installation
          WT-2948 simplify error handling by making epoch time return never fail
          WT-2950 Inserting multi-megabyte values can cause large in-memory pages
          Branch: master
          https://github.com/mongodb/mongo/commit/448f5b43d249dd090da1d79b82c9ab1b742e4f38

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 6a31c2118cce88c68281eda2ca9ab1df915a2773 from branch mongodb-3.4 ref: fc0e7abe82..6a31c2118c for: 3.4.0 WT-2831 Skip creating a checkpoint if there have been no changes WT-2858 rename wtperf's CONFIG structure WT-2895 Reduce the runtime of make check testing with disable long WT-2907 Bug in Java ConcurrentCloseTest case WT-2917 split wtperf's configuration into per-database and per-run parts WT-2920 New eviction statistics WT-2931 Configure default in-memory dirty cache usage lower WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration WT-2933 Fix a race between named snapshots and checkpoints WT-2937 test_inmem01 aborts due to stuck cache WT-2938 Assembly files should end in .sx, not .S WT-2941 Improve test/format to use faster key-generation functions WT-2942 verbose strings don't need newline WT-2946 dist/s_docs incompatible with OS X Xcode installation WT-2948 simplify error handling by making epoch time return never fail WT-2950 Inserting multi-megabyte values can cause large in-memory pages Branch: master https://github.com/mongodb/mongo/commit/448f5b43d249dd090da1d79b82c9ab1b742e4f38
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081)
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2950 Free all reconciliation memory between calls to eviction. (#3081) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/6775c3401a4e51aeff02c0f369ba7a8a0b9af633
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2

          ref: 9cf2f89d6d..b11ed312ce
          for: 3.2.11

          WT-1592 Dump detailed cache information via statistics
          WT-2403 Enhance random cursor implementation for LSM trees
          WT-2831 Skip creating a checkpoint if there have been no changes
          WT-2858 rename wtperf's CONFIG structure
          WT-2880 Add support for Zstandard compression
          WT-2895 Reduce the runtime of make check testing with disable long
          WT-2904 Fix a bug where the reported checkpoint size could be many times data size
          WT-2907 Bug in Java ConcurrentCloseTest case
          WT-2917 split wtperf's configuration into per-database and per-run parts
          WT-2920 Add statistic tracking application thread cache maintenance time
          WT-2931 Configure default in-memory dirty cache usage lower
          WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration
          WT-2933 Fix a race between named snapshots and checkpoints
          WT-2937 test_inmem01 aborts due to stuck cache
          WT-2938 Assembly files should end in .sx, not .S
          WT-2941 Improve test/format to use faster key-generation functions
          WT-2942 verbose strings don't need newline
          WT-2946 dist/s_docs incompatible with OS X Xcode installation
          WT-2948 simplify error handling by making epoch time return never fail
          WT-2949 Add an option to wtperf to not close connection on shutdown
          WT-2950 Inserting multi-megabyte values can cause large in-memory pages
          WT-2954 Inserting multi-megabyte values can cause large in-memory pages
          WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks
          WT-2956 utility tests -h option is always overridden by the default setup
          WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized
          WT-2963 Race setting max_entries during eviction
          WT-2965 test_wt2323_join_visibility can hang on OSX
          WT-2974 lint
          WT-2976 Add a statistic tracking how long application threads spend doing I/O
          WT-2977 Csuite LSM Random test can occasionally fail
          WT-2985 Race during checkpoint can cause a core dump
          WT-2987 Fix a bug where opening a cursor on an incomplete table drops core
          WT-2988 __wt_epoch potentially returns garbage values.
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2 ref: 9cf2f89d6d..b11ed312ce for: 3.2.11 WT-1592 Dump detailed cache information via statistics WT-2403 Enhance random cursor implementation for LSM trees WT-2831 Skip creating a checkpoint if there have been no changes WT-2858 rename wtperf's CONFIG structure WT-2880 Add support for Zstandard compression WT-2895 Reduce the runtime of make check testing with disable long WT-2904 Fix a bug where the reported checkpoint size could be many times data size WT-2907 Bug in Java ConcurrentCloseTest case WT-2917 split wtperf's configuration into per-database and per-run parts WT-2920 Add statistic tracking application thread cache maintenance time WT-2931 Configure default in-memory dirty cache usage lower WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration WT-2933 Fix a race between named snapshots and checkpoints WT-2937 test_inmem01 aborts due to stuck cache WT-2938 Assembly files should end in .sx, not .S WT-2941 Improve test/format to use faster key-generation functions WT-2942 verbose strings don't need newline WT-2946 dist/s_docs incompatible with OS X Xcode installation WT-2948 simplify error handling by making epoch time return never fail WT-2949 Add an option to wtperf to not close connection on shutdown WT-2950 Inserting multi-megabyte values can cause large in-memory pages WT-2954 Inserting multi-megabyte values can cause large in-memory pages WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks WT-2956 utility tests -h option is always overridden by the default setup WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized WT-2963 Race setting max_entries during eviction WT-2965 test_wt2323_join_visibility can hang on OSX WT-2974 lint WT-2976 Add a statistic tracking how long application threads spend doing I/O WT-2977 Csuite LSM Random test can occasionally fail WT-2985 Race during checkpoint can cause a core dump WT-2987 Fix a bug where opening a cursor on an incomplete table drops core WT-2988 __wt_epoch potentially returns garbage values. Branch: v3.2 https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                22 weeks ago
                Date of 1st Reply:

                  Agile