[SERVER-12886] Make non-fastmod updates perform work proportional to the update size Created: 25/Feb/14 Updated: 06/Dec/22 Resolved: 14/Sep/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | MMAPv1, Performance, Write Ops |
| Affects Version/s: | 2.6.0-rc0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | James Blackburn | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Won't Fix | Votes: | 2 |
| Labels: | mmapv1, pull-request | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Assigned Teams: |
Storage Execution
|
||||
| Participants: | |||||
| Case: | (copied to CRM) | ||||
| Description |
|
Currently non-fastmod updates generate I/O proportional to the size of the document (which is pathological for large documents and multi-updates). See the linked and other related tickets to high write I/O cause by update amplification. A proposed fix is two-fold:
I believe 2 is OK, because:
This is related, but different, to: I've attempted implementations of 1 + 2 which seem to give 2 orders of magnitude improvement to performance of the benchmarks on the linked ticket:
|
| Comments |
| Comment by Benety Goh [ 16/Jun/14 ] | |||||||||||||||||||||||||||
|
Hi James, We've taken another look at your pull request. Unfortunately, due to recent changes in the storage layer, your commit no longer rebases cleanly on top of master (commit 82fadf3). Even after resolving the merge conflicts, we encountered a crash in the dbtest test suite: Command line for building and running test:
Test output
We are still interested in the changes proposed in the pull request and will keep this SERVER ticket open. The existing pull request #646 will be closed. A new pull request rebased on a more recent commit would have to be submitted that does not contain the re-ordering logic. Regards, | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 03/Mar/14 ] | |||||||||||||||||||||||||||
|
Thanks for taking the time to review Andrew! Regarding your points:
Removing the re-order breaks the optimization and returns the performance to the baseline. This is straightforward to see in the attached version_store.py: by moving the parent field above or below the 4MB data. As the 4MB chunk is made or random data, having the parent array before the data causes a full-rewrite, whereas having it after causes just the last page to be re-written. I think not optimising in the DB will require people to know / decide in advance which fields are likely to change, and forces them to use SON objects with fixed ordering - arrays, and nested documents at the end. This exposes an implementation detail: changing the size of an early field causes the remainder of the document to be re-written. Adding additional config options would also seem unnecessary if there really isn't a use-case for switching this option off. | |||||||||||||||||||||||||||
| Comment by Andrew Morrow (Inactive) [ 28/Feb/14 ] | |||||||||||||||||||||||||||
|
Hi - Thank you for the pull requests. We are currently in code freeze for the upcoming 2.6.0 release, so this change (or something like it) cannot be merged until the master branch re-opens for 2.7 changes and the 2.6 branch opens for 2.6.1 changes. Here are some comments on your submission:
Would you be willing to re-run your benchmarks without the document reordering behavior, and post your results? Overall, these are good ideas and we appreciate your taking the time to share your implementation of them with us. Thanks, | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 28/Feb/14 ] | |||||||||||||||||||||||||||
|
Running the python mongostat.py --journal script, you see: MongoDB:
You see up to 46 operations happening per second vs 2-4. | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 28/Feb/14 ] | |||||||||||||||||||||||||||
|
Benchmark with 4MB documents:
In this benchmark this patch makes MongoDB 10-15x faster for updates than the default build and Toku build. | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 27/Feb/14 ] | |||||||||||||||||||||||||||
|
Attached is a python mongostat which can be used for printing journal and DB files write stats
These changes also improves the sysbench benchmark by 50% for large documents: (and doesn't seem to penalize small document updates). | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 26/Feb/14 ] | |||||||||||||||||||||||||||
|
Added a performance test demonstrating the difference. Before the changes altering an indexed field, or an array field causes up to 10MB of I/O. As well as reducing I/O this appears to reduce time in the writeLock by ~30%. Pull request submitted: | |||||||||||||||||||||||||||
| Comment by James Blackburn [ 26/Feb/14 ] | |||||||||||||||||||||||||||
|
Patches related to this change are in the following branch: Difference to master: |