[SERVER-32707] WiredTiger performance with the insert benchmark Created: 15/Jan/18 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.6.0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Mark Callaghan | Assignee: | Backlog - Performance Team |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Product Performance
|
||||||||
| Participants: | |||||||||
| Description |
|
I ran the insert benchmark for WiredTiger in MongoDB 3.6.0 and summarize the problems I see here. For more details on the insert benchmark including a link to the source see this link. The overview of the insert benchmark is: I have seen this in all of the major MongoDB releases (3.0, 3.2, 3.4) and now 3.6.0. This time I archived diagnostic data. I tried 9 mongo.conf variations for 4 test configurations:
The test server has 24 cores and 48 HW threads. Hyperthreading is enabled. For the in-memory benchmarks the server has 256gb of RAM. For the IO-bound benchmarks the server has 50gb of RAM. The server also has 2 or 3 fast PCIe-based SSDs. The template for mongo.conf for this in-memory benchmarks is below and comments at the end explain the 9 variations:
The mongo.conf template for the IO-bound tests is below. The big difference from the configuration above is cacheSizeGB is reduces from 180 to 10. I won't paste mongo.conf for the test that used compression, but the change from the template below is obvious.
|
| Comments |
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
The data below has metrics from vmstat and iostat. Some are normalized by the insert rate. This makes it easy to see why the average insert rates are better for InnoDB than for WiredTiger:
These are the metrics for inMemory-16 with configuration 1 for WiredTiger.
These are the metrics for ioBound-none with configuration 1 for WiredTiger
| ||||||||||
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
The uploaded diagnostic.data files are collected across all of the tests – load, scan, read-write | ||||||||||
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
For the ioBound-none test and configuration 1 | ||||||||||
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
For the inMemory-16 test and configuration 9 | ||||||||||
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
For the inMemory-16 test and configuration 7 | ||||||||||
| Comment by Mark Callaghan [ 16/Jan/18 ] | ||||||||||
|
For the inMemory-16 test and configuration 1 | ||||||||||
| Comment by Ramon Fernandez Marina [ 16/Jan/18 ] | ||||||||||
|
Hi mdcallag, if you still have the contents of the diagnostic.data directory for your tests, will you please upload them to the ticket? Thanks, | ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
Using my helper script a typical command line to run all of the tests (load, scan, read-write) is:
| ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
For the ioBound-zlib test (IO-bound, 16 clients, client per collection, zlib compression). The average insert rates per configuration are: And the worst-case response time in seconds per configuration: I won't compare this with InnoDB because InnoDB compression isn't ready for prime time. | ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
I didn't test the IO-bound configuration with 1 collection and 16 concurrent clients because that takes too long to get results for the nine variations for mongo.conf. From previous tests the average insert rate for WiredTiger drops from ~15,000/second to ~3500/second in the test that uses 1 collection. For the IO-bound setup with 1 collection and WiredTiger in MongoDB 3.4.6 I get
| ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
For ioBound-none (io-bound, no compression) Average insert rate per configuration Worst-case response time for an insert during the load per configuration: For comparison, InnoDB in MySQL 5.7 gets
I assume the worst-case stalls here are better than for the in-memory tests because the in-memory tests sustain much higher average insert rates, so there is more stress. My comparison with InnoDB isn't done to cast FUD at WiredTiger. We spent a long time fixing InnoDB stalls from workloads like this. This is a hard problem. | ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
Next up is inMemory-16 (in-memory, 16 clients, collection per client). The results here are similar to the results for inMemory-1. Average insert rates during the load per configuration. Configuration 9 has the best rate: This is the worst-case response time in seconds per insert during the load. Configuration 9 has the best of the worst-cases, about 7 seconds. | ||||||||||
| Comment by Mark Callaghan [ 15/Jan/18 ] | ||||||||||
|
First up is inMemory-1 (in-memory, 1 collection, 16 clients). This is the average insert rate per configuration: And the worst-case response time per insert in seconds during the load. The best configuration suffers a stall for at least 7 seconds. For comparison with modern InnoDB (5.7, 8.0), although the MySQL tests insert 500m rows while I limit MongoDB to 250m because it uses much more space:
So with a good configuration, WiredTiger can get insert rates similar to InnoDB but stalls for WiredTiger are much worse. |