[SERVER-32424] Use WiredTiger cursor caching Created: 20/Dec/17  Updated: 30/Oct/23  Resolved: 13/Apr/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 3.6.10, 3.7.4

Type: Improvement Priority: Major - P3
Reporter: Donald Anderson Assignee: Donald Anderson
Resolution: Fixed Votes: 1
Labels: SEKB
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-33741 Enhance capped collection deletion of... Closed
Problem/Incident
Related
related to SERVER-39355 Collection drops can block the server... Closed
is related to SERVER-33122 add option to limit WiredTiger cursor... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6
Sprint: Storage 2018-02-26, Storage Non-NYC 2018-04-09, Storage Non-NYC 2018-04-23
Participants:
Linked BF Score: 0

 Description   

As part of WT-1228, we can enable cursor caching for WiredTiger in the integration layer. Originally, we intended to remove the cursor caching done in the integration layer. To get roughly equivalent performance, we should keep the caching at the integration layer (with a much lower count of cursors), and when a session is released, all cursors in the session will be closed. Closing the cursors will cache them in WiredTiger, so that reopens will be quick. Thus, this is called a hybrid scheme. It has the benefit of the original plan that we should be able to immediately drop or verify collections that are no longer in use.



 Comments   
Comment by Githook User [ 04/Jan/19 ]

Author:

{'username': 'lukech', 'email': 'luke.chen@mongodb.com', 'name': 'Luke Chen'}

Message: SERVER-32424 Use hybrid cursor caching by default.
This mechanism is a hybrid between the current way of doing cursor caching
in the integration layer and doing cursor caching in WiredTiger.
This gives the best compromise between allowing exclusive operations
(drop/verify) to succeed, and good performance for typical workloads.
The old cursor caching scheme, as well as one that caches purely in
WiredTiger, are available as options when starting the server.
Added a performance improvement for capped collections, especially under
the new cursor caching schemes.

(cherry picked from commit c0d20e4fb98626143d772bb70a950b277685bbbd)
Branch: v3.6
https://github.com/mongodb/mongo/commit/e9e710e6050d9025250f2ba7cb4a2ffcd711b1e1

Comment by Githook User [ 13/Apr/18 ]

Author:

{'email': 'dda@mongodb.com', 'name': 'Don Anderson', 'username': 'ddanderson'}

Message: SERVER-32424 Use hybrid cursor caching by default.
This mechanism is a hybrid between the current way of doing cursor caching
in the integration layer and doing cursor caching in WiredTiger.
This gives the best compromise between allowing exclusive operations
(drop/verify) to succeed, and good performance for typical workloads.
The old cursor caching scheme, as well as one that caches purely in
WiredTiger, are available as options when starting the server.
Added a performance improvement for capped collections, especially under
the new cursor caching schemes.
Branch: master
https://github.com/mongodb/mongo/commit/c0d20e4fb98626143d772bb70a950b277685bbbd

Comment by Alexander Gorrod [ 10/Apr/18 ]

donald.anderson I believe all the pieces are in place to make this change now. Could you proceed with:

Changing the default behavior so that:

  • The WiredTiger cursor cache is turned on in MongoDB.
  • The MongoDB cursor cache is turned off.

Also adding a method for MongoDB users to revert to the old behavior via a configuration option. I'm hoping there is a way to tie that into the new wiredTigerCursorCacheSize option added in SERVER-33122. Maybe changing the default value of that to 0, and having any non-zero value disable the WiredTiger cursor cache? It's also a signed value, so we could use 0 to disable all caching, and a value of -1 to enable using the WiredTiger cache, but that's pretty non-intuitive.

Comment by Alexander Gorrod [ 21/Mar/18 ]

The WiredTiger cursor cache has been disabled via a configuration option in SERVER-34024, the final change here should re-enable the WiredTiger cursor cache.

Comment by Alexander Gorrod [ 13/Mar/18 ]

Given the scope of this change, and how sensitive some workloads are to cursor caching, I'm inclined to keep the existing cursor cache schema as a configurable startup option. That will give a reasonable escape valve if we discover any cases where the new caching scheme causes degraded performance characteristics.

Hopefully that can tie into the work done in SERVER-33122

Comment by Alexander Gorrod [ 08/Mar/18 ]

When implementing WT-3938 we encountered an inefficiency in the implementation of capped collections in WiredTiger, where we were opening 4 cursors. The changes in this ticket should include an optimization that reduces the number of cursors used in capped collection size maintenance.

Comment by Alexander Gorrod [ 22/Feb/18 ]

donald.anderson Is this a duplicate of SERVER-33378?

Generated at Thu Feb 08 04:30:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.