Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2932

Allow applications to selectively ignore cache limit with in-memory configuration

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc1
    • Labels:
      None
    • # Replies:
      6
    • Last comment by Customer:
      true

      Description

      The in-memory storage engine currently returns WT_CACHE_FULL if an application attempts to do any insert or update operation when the cache is at capacity.

      There are use cases where applications might want to allow specific operations to complete even though the cache is full. The particular use case in mind is an application that tracks metadata in a WiredTiger table and requires that inserts/updates to the metadata are allowed for correctness.

        Issue Links

          Activity

          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          We have discussed two alternative options:

          • Allow certain tables to be marked as excluded from cache checks at creation time.
          • Allow a cursor level flag that marks operations on the cursor as excluded.

          The table level option is preferred because it gives a simpler API, that is easier to document and therefore to understand.

          I think the option will be to WT_SESSION::create called in_memory_ignore_cache_size=true, which will be a boolean default to false.

          Show
          alexander.gorrod Alexander Gorrod added a comment - We have discussed two alternative options: Allow certain tables to be marked as excluded from cache checks at creation time. Allow a cursor level flag that marks operations on the cursor as excluded. The table level option is preferred because it gives a simpler API, that is easier to document and therefore to understand. I think the option will be to WT_SESSION::create called in_memory_ignore_cache_size=true , which will be a boolean default to false.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069)

          Only available for in-memory configurations, designed to be used for
          applications that maintain a small amount of content that is required
          for correctness.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069) Only available for in-memory configurations, designed to be used for applications that maintain a small amount of content that is required for correctness. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069)

          Only available for in-memory configurations, designed to be used for
          applications that maintain a small amount of content that is required
          for correctness.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069) Only available for in-memory configurations, designed to be used for applications that maintain a small amount of content that is required for correctness. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: 6a31c2118cce88c68281eda2ca9ab1df915a2773 from branch mongodb-3.4

          ref: fc0e7abe82..6a31c2118c
          for: 3.4.0

          WT-2831 Skip creating a checkpoint if there have been no changes
          WT-2858 rename wtperf's CONFIG structure
          WT-2895 Reduce the runtime of make check testing with disable long
          WT-2907 Bug in Java ConcurrentCloseTest case
          WT-2917 split wtperf's configuration into per-database and per-run parts
          WT-2920 New eviction statistics
          WT-2931 Configure default in-memory dirty cache usage lower
          WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration
          WT-2933 Fix a race between named snapshots and checkpoints
          WT-2937 test_inmem01 aborts due to stuck cache
          WT-2938 Assembly files should end in .sx, not .S
          WT-2941 Improve test/format to use faster key-generation functions
          WT-2942 verbose strings don't need newline
          WT-2946 dist/s_docs incompatible with OS X Xcode installation
          WT-2948 simplify error handling by making epoch time return never fail
          WT-2950 Inserting multi-megabyte values can cause large in-memory pages
          Branch: master
          https://github.com/mongodb/mongo/commit/448f5b43d249dd090da1d79b82c9ab1b742e4f38

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 6a31c2118cce88c68281eda2ca9ab1df915a2773 from branch mongodb-3.4 ref: fc0e7abe82..6a31c2118c for: 3.4.0 WT-2831 Skip creating a checkpoint if there have been no changes WT-2858 rename wtperf's CONFIG structure WT-2895 Reduce the runtime of make check testing with disable long WT-2907 Bug in Java ConcurrentCloseTest case WT-2917 split wtperf's configuration into per-database and per-run parts WT-2920 New eviction statistics WT-2931 Configure default in-memory dirty cache usage lower WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration WT-2933 Fix a race between named snapshots and checkpoints WT-2937 test_inmem01 aborts due to stuck cache WT-2938 Assembly files should end in .sx, not .S WT-2941 Improve test/format to use faster key-generation functions WT-2942 verbose strings don't need newline WT-2946 dist/s_docs incompatible with OS X Xcode installation WT-2948 simplify error handling by making epoch time return never fail WT-2950 Inserting multi-megabyte values can cause large in-memory pages Branch: master https://github.com/mongodb/mongo/commit/448f5b43d249dd090da1d79b82c9ab1b742e4f38
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069)

          Only available for in-memory configurations, designed to be used for
          applications that maintain a small amount of content that is required
          for correctness.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-2932 Add a configuration option allowing tables to ignore cache limits (#3069) Only available for in-memory configurations, designed to be used for applications that maintain a small amount of content that is required for correctness. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/e389a9019a467a9a5127ada42e0e1bb36df29790
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2

          ref: 9cf2f89d6d..b11ed312ce
          for: 3.2.11

          WT-1592 Dump detailed cache information via statistics
          WT-2403 Enhance random cursor implementation for LSM trees
          WT-2831 Skip creating a checkpoint if there have been no changes
          WT-2858 rename wtperf's CONFIG structure
          WT-2880 Add support for Zstandard compression
          WT-2895 Reduce the runtime of make check testing with disable long
          WT-2904 Fix a bug where the reported checkpoint size could be many times data size
          WT-2907 Bug in Java ConcurrentCloseTest case
          WT-2917 split wtperf's configuration into per-database and per-run parts
          WT-2920 Add statistic tracking application thread cache maintenance time
          WT-2931 Configure default in-memory dirty cache usage lower
          WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration
          WT-2933 Fix a race between named snapshots and checkpoints
          WT-2937 test_inmem01 aborts due to stuck cache
          WT-2938 Assembly files should end in .sx, not .S
          WT-2941 Improve test/format to use faster key-generation functions
          WT-2942 verbose strings don't need newline
          WT-2946 dist/s_docs incompatible with OS X Xcode installation
          WT-2948 simplify error handling by making epoch time return never fail
          WT-2949 Add an option to wtperf to not close connection on shutdown
          WT-2950 Inserting multi-megabyte values can cause large in-memory pages
          WT-2954 Inserting multi-megabyte values can cause large in-memory pages
          WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks
          WT-2956 utility tests -h option is always overridden by the default setup
          WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized
          WT-2963 Race setting max_entries during eviction
          WT-2965 test_wt2323_join_visibility can hang on OSX
          WT-2974 lint
          WT-2976 Add a statistic tracking how long application threads spend doing I/O
          WT-2977 Csuite LSM Random test can occasionally fail
          WT-2985 Race during checkpoint can cause a core dump
          WT-2987 Fix a bug where opening a cursor on an incomplete table drops core
          WT-2988 __wt_epoch potentially returns garbage values.
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2 ref: 9cf2f89d6d..b11ed312ce for: 3.2.11 WT-1592 Dump detailed cache information via statistics WT-2403 Enhance random cursor implementation for LSM trees WT-2831 Skip creating a checkpoint if there have been no changes WT-2858 rename wtperf's CONFIG structure WT-2880 Add support for Zstandard compression WT-2895 Reduce the runtime of make check testing with disable long WT-2904 Fix a bug where the reported checkpoint size could be many times data size WT-2907 Bug in Java ConcurrentCloseTest case WT-2917 split wtperf's configuration into per-database and per-run parts WT-2920 Add statistic tracking application thread cache maintenance time WT-2931 Configure default in-memory dirty cache usage lower WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration WT-2933 Fix a race between named snapshots and checkpoints WT-2937 test_inmem01 aborts due to stuck cache WT-2938 Assembly files should end in .sx, not .S WT-2941 Improve test/format to use faster key-generation functions WT-2942 verbose strings don't need newline WT-2946 dist/s_docs incompatible with OS X Xcode installation WT-2948 simplify error handling by making epoch time return never fail WT-2949 Add an option to wtperf to not close connection on shutdown WT-2950 Inserting multi-megabyte values can cause large in-memory pages WT-2954 Inserting multi-megabyte values can cause large in-memory pages WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks WT-2956 utility tests -h option is always overridden by the default setup WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized WT-2963 Race setting max_entries during eviction WT-2965 test_wt2323_join_visibility can hang on OSX WT-2974 lint WT-2976 Add a statistic tracking how long application threads spend doing I/O WT-2977 Csuite LSM Random test can occasionally fail WT-2985 Race during checkpoint can cause a core dump WT-2987 Fix a bug where opening a cursor on an incomplete table drops core WT-2988 __wt_epoch potentially returns garbage values. Branch: v3.2 https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                29 weeks, 6 days ago
                Date of 1st Reply: