[SERVER-57181] Evict server should be parallelized Created: 17/May/21  Updated: 24/Jun/21  Resolved: 24/Jun/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: peng zhenyi Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2021-05-21-17-28-30-711.png     PNG File image-2021-05-21-17-30-18-131.png    
Operating System: ALL
Participants:

 Description   

Evict-server is single-threaded in wiredtiger. Sometimes it becomes the bottleneck under heavy write workloads. As WT cache-used and cache-dirty get too high, app threads are stuck in evict-worker process but won't help.

Single-threaded evict-server may lead 2 problems:

  1. Reduce write performance even in multi-core machines.
  2. The related cpu core usage spikes to 100% when evict-server is busy.


 Comments   
Comment by Dmitry Agranat [ 24/Jun/21 ]

Hi pengzhenyi,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 09/Jun/21 ]

Hi pengzhenyi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please upload the requested information?

Thanks,
Dima

Comment by Dmitry Agranat [ 25/May/21 ]

Hi pengzhenyi, to understand the reported issue, we'd like to review the workload you are executing.

Would you please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Please mention the exact timestamp and timezone for the start/stop of the event in question.

Thanks,
Dima

Comment by peng zhenyi [ 21/May/21 ]

HI, brian.lane

 

I found this issue in  v3.2 (r3.2.10) and v4.0 (r4.0.3)  which under heavy-update workloads. 

I ran 'pstack' of the thread which consumed the entire single cpu core, and conformed that it was the evict-server task.

I also read the latest WT release code (WT 10.0.0), and found that only one thread can be evict-server which gets the 'evict_pass_lock'.

 

Here are my questions

  1. Evict-server is single-threaded, Is there any reason about this?
  2. Evict-server may cause cpu usage spikes, and even become the bottleneck of WT. Is there any way to solve this? I tried to increase more evict threads, but did not help.
  3. Does WT community have any plans to make evict-server multi-threaded?
Comment by Brian Lane [ 18/May/21 ]

HI pengzhenyi,

Thanks for reaching out with this issue.

Could you give us more details about your workload and how often you are experiencing this behavior?

Generated at Thu Feb 08 05:41:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.