[SERVER-31704] Periodic drops in throughput during checkpoints while waiting on schema lock Created: 24/Oct/17  Updated: 17/Aug/23  Resolved: 24/Sep/21

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.9
Fix Version/s: 4.3 Desired

Type: Bug Priority: Major - P3
Reporter: Kelsey Schubert Assignee: Brian Lane
Resolution: Duplicate Votes: 13
Labels: customer-mgmt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File checkpoint_lock_4.2.7.png     File diagnostic.tar.gz     File mix.js     PNG File no_checkpoint.png     PNG File periodic_drops.png    
Issue Links:
Duplicate
is duplicated by WT-5479 Checkpoint schema lock can cause long... Closed
Related
related to SERVER-32890 Background index creation sometimes b... Closed
related to WT-6421 Avoid parsing metadata checkpoint for... Closed
is related to SERVER-40773 Slow queries every 80s under moderate... Closed
Operating System: ALL
Steps To Reproduce:

Run mix.js
Set the following parameter:

wiredTigerEngineRuntimeConfig:"file_manager=(close_handle_minimum=100,close_idle_time=300,close_scan_interval=30)"

Participants:
Case:

 Description   

Periodic drops in throughput can be observed during checkpoints while running with an artificial workload and specific WiredTiger tuning parameters enabled.

I've attached diagnostic.tar.gz that shows this behavior. At around 2017-10-20T20:10:08.587Z the following command was executed:

db.adminCommand({setParameter: 1, wiredTigerEngineRuntimeConfig:"file_manager=(close_handle_minimum=100,close_idle_time=300,close_scan_interval=30)"});

After reaching steady state, we see the following:

Additionally, from perf, we can see that after making this parameter change, a single cpu core (12.5% of the total) is consistently fully utilized running eviction:

start_thread+0xffff017a586c80c5
__wt_thread_run+0xffff5458d1e2e016
__wt_evict_thread_run+0xffff5458d1e2e951



 Comments   
Comment by Eric Milkie [ 15/Aug/20 ]

WT-6421 was not backported to 4.2 yet. The fixVersion numbers on the ticket that do not have "WT" in them represent the MongoDB releases that contain the ticket work.

Comment by Chad Kreimendahl [ 14/Aug/20 ]

Will WT-6421 be implemented as a partial resolution to this? What versions can we expect it to be in? I saw it was backported to 4.2 and possibly 4.0, but don't see where that maps into a specific release version of 4.2 or 4.0

Comment by Ralf Strobel [ 16/Jun/20 ]

WT-5479 which I created after our own debugging may essentially be a duplicate of this issue and contains some more depictions and benchmarks by affected users.

Comment by Andy Pang [ 10/Jun/20 ]

I don't see the in question macro listed here. Here are our patches:

https://github.com/apang-ns/mongo/commit/1aea123a49f6eed873694a8d5dd9a46f903a3dde

https://github.com/apang-ns/mongo/commit/b23b9b1a0a079e9835a3e16a91ddaaaaacc83014

Percona has a better version of this patch they included in their 3.6 release train if you're looking to experiment: https://www.percona.com/doc/percona-server-for-mongodb/3.6/release_notes/3.6.18-6.0.html

As mentioned in this thread, WT-5042 makes reading/writing checkpoint metadata faster in 3.6.17+, so you'd see improvement even if you did not increase the hash array sizes with the config that is introduced (i.e. big_hash_array_size, session_dhhash_size).

Comment by Firass Almiski [ 09/Jun/20 ]

So which of these options did you tweak Andy? 

https://source.wiredtiger.com/mongodb-3.4/group__wt.html#ga8ca567b2908997280e4f0a20b80b358b

 

I'd like to give that a try. I'm seeing issues that I believe are related to dhandle as well. 

Comment by Andy Pang [ 09/Jun/20 ]

For high data handle count workloads, we've found that WT spends 50% of checkpoint prepare time traversing dhandles, and during this time requests from app threads are not serviced.

Our fix was to increase the hash array size for dhandle (currently hardcoded to in WT 512 buckets) for more efficient lookup. We were able to improve performance by 50%+ during checkpoint prepare (the time when the schema lock is held) by eliminating the traversal time using the existing hash mechanism.

Comment by Brian Lane [ 24/Mar/20 ]

Hi falmiski@creativeradicals.com,

There currently isn't an ETA for this one - I had wanted to spend some time digging into it before the upcoming 4.4 release, but I expect it will need to wait until after that release is finished. Could you give me more details on which version you are using and the issue you are experiencing?

Comment by Firass Almiski [ 23/Mar/20 ]

Requesting update, once again

Comment by Firass Almiski [ 18/Mar/20 ]

ETA of this fix? Please bump the priority if at all possible...

Comment by Brian Lane [ 18/Feb/20 ]

dmitry.agranat could you perhaps re-run this with a version that has WT-5042 included. We have seen up to 50% improvement in some cases for lock duration. Wonder if we can resolve this one now.

Comment by Kelsey Schubert [ 01/Nov/17 ]

I don't believe this is new as I see similar behavior in MongoDB 3.4.4. Unfortunately, given the nature of the workload, I wasn't able to determine whether 3.4.0 is similarly affected.

Comment by Alexander Gorrod [ 24/Oct/17 ]

anonymous.user Do you know if this symptom is new, or if it was present in previous releases of MongoDB?

Comment by Kelsey Schubert [ 24/Oct/17 ]

alex.komyagin encountered this issue during some testing around tuning workloads with many collections.

Generated at Thu Feb 08 04:27:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.