[SERVER-9802] Single-threaded journal compression becomes a bottleneck when using "durable" writes Created: 29/May/13 Updated: 06/Dec/22 Resolved: 14/Sep/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | MMAPv1, Performance, Storage |
| Affects Version/s: | 2.4.3 |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Dmitry Naumov | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Won't Fix | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows 7 Professional SP1 x64 |
||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
When multiple threads save documents in "durable" fashion (I mean getLastError: 1, j: true) they all wait untill one thread running durThread() method persist these changes. Also this thread does compression, which becomes a bottleneck. More details are here https://jira.mongodb.org/browse/SERVER-9754?focusedCommentId=347003&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347003 |
| Comments |
| Comment by Mark Callaghan [ 21/Nov/14 ] |
|
I don't think this is needed given the arrival of the engine API and WiredTiger in 2.8. I prefer that the core team at MongoDB focus on other features. |
| Comment by Dmitry Naumov [ 21/Nov/14 ] |
|
After disabling journal compression is extracted to separate item, now we can discuss only the case when journalling is enabled and compression is turned on. Let me rephrase title of this issue: does it really makes sense to compress buffers in one thread while other writers are waiting this compression to finish? As far as I remember each buffer is compressed individually, so why don't let writer thread compress it before putting in flushing queue? In this case journalling thread responsibility narrows to flush these already compressed buffers to disk and release waiting writers. In other words - in this case we can leverage multi-core CPU. |
| Comment by Andy Schwerin [ 09/Apr/14 ] |
|
I've scheduled |
| Comment by Dmitry Naumov [ 25/Mar/14 ] |
|
Mark, |
| Comment by Mark Callaghan [ 25/Mar/14 ] |
|
Forcing journal file preallocation and recycling even for flash storage can help – https://jira.mongodb.org/browse/SERVER-13346 |
| Comment by Mark Callaghan [ 25/Mar/14 ] |
|
Created https://jira.mongodb.org/browse/SERVER-13345 asking for an option to disable journal compression. I have workloads that get better throughput with it disabled. |
| Comment by Mark Callaghan [ 25/Mar/14 ] |
|
Current code pads journal writes to a multiple of 8kb. For j:1 workloads with small changes padding to 4kb will help . See https://jira.mongodb.org/browse/SERVER-13344 |
| Comment by Mark Callaghan [ 25/Mar/14 ] |
|
I don't think the code should try to compress journal writes that are already less than 8kb – https://jira.mongodb.org/browse/SERVER-13343 |
| Comment by Dmitry Naumov [ 10/Jun/13 ] |
|
Should I provide some benchmarks or something? This issue was hidden by |