[SERVER-9802] Single-threaded journal compression becomes a bottleneck when using "durable" writes Created: 29/May/13  Updated: 06/Dec/22  Resolved: 14/Sep/18

Status: Closed
Project: Core Server
Component/s: MMAPv1, Performance, Storage
Affects Version/s: 2.4.3
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Dmitry Naumov Assignee: Backlog - Storage Execution Team
Resolution: Won't Fix Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7 Professional SP1 x64


Issue Links:
Related
is related to SERVER-13343 Don't compress journal writes that ar... Closed
is related to SERVER-13344 Pad journal writes to 4k not 8k Closed
is related to SERVER-13345 Provide an option to disable journal ... Closed
Assigned Teams:
Storage Execution
Participants:

 Description   

When multiple threads save documents in "durable" fashion (I mean getLastError: 1, j: true) they all wait untill one thread running durThread() method persist these changes. Also this thread does compression, which becomes a bottleneck. More details are here https://jira.mongodb.org/browse/SERVER-9754?focusedCommentId=347003&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347003



 Comments   
Comment by Mark Callaghan [ 21/Nov/14 ]

I don't think this is needed given the arrival of the engine API and WiredTiger in 2.8. I prefer that the core team at MongoDB focus on other features.

Comment by Dmitry Naumov [ 21/Nov/14 ]

After disabling journal compression is extracted to separate item, now we can discuss only the case when journalling is enabled and compression is turned on.

Let me rephrase title of this issue: does it really makes sense to compress buffers in one thread while other writers are waiting this compression to finish? As far as I remember each buffer is compressed individually, so why don't let writer thread compress it before putting in flushing queue? In this case journalling thread responsibility narrows to flush these already compressed buffers to disk and release waiting writers.

In other words - in this case we can leverage multi-core CPU.

Comment by Andy Schwerin [ 09/Apr/14 ]

I've scheduled SERVER-13343, SERVER-13344 and SERVER-13345 to the tail half of the 2.7/2.8 dev cycle, time permitting. This should alleviate this problem. Given that journal writing will always be one of the activities that can delay responding to durable writes, I'm not entirely certain what the criteria for resolving this particular ticket can be. Until we agree on the success criteria, I'm going to mark this "Needs Further Definition". It's not a perfect categorization, but it's more accurate than "Needs Triage".

Comment by Dmitry Naumov [ 25/Mar/14 ]

Mark,
I think disabling journal compression is a good thing to start. At least it should start scaling on cpu cores.

Comment by Mark Callaghan [ 25/Mar/14 ]

Forcing journal file preallocation and recycling even for flash storage can help – https://jira.mongodb.org/browse/SERVER-13346

Comment by Mark Callaghan [ 25/Mar/14 ]

Created https://jira.mongodb.org/browse/SERVER-13345 asking for an option to disable journal compression. I have workloads that get better throughput with it disabled.

Comment by Mark Callaghan [ 25/Mar/14 ]

Current code pads journal writes to a multiple of 8kb. For j:1 workloads with small changes padding to 4kb will help . See https://jira.mongodb.org/browse/SERVER-13344

Comment by Mark Callaghan [ 25/Mar/14 ]

I don't think the code should try to compress journal writes that are already less than 8kb – https://jira.mongodb.org/browse/SERVER-13343

Comment by Dmitry Naumov [ 10/Jun/13 ]

Should I provide some benchmarks or something? This issue was hidden by SERVER-9754, but now it's negative effect should be more visible.

Generated at Thu Feb 08 03:21:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.