[SERVER-17966] Mixed read/write workload using zlib unexpectedly outperforms snappy on Windows Created: 09/Apr/15 Updated: 25/Aug/15 Resolved: 25/Aug/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Nick Judson | Assignee: | Eitan Klein |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | Windows | ||||||||
| Participants: | |||||||||
| Description |
|
Submitting on behalf of a user. Will fill in details shortly ~~~~~~~~ Hardware used: Test results:
|
| Comments |
| Comment by Ramon Fernandez Marina [ 25/Aug/15 ] | |
|
Reopening to adjust resolution. | |
| Comment by Eitan Klein [ 25/Aug/15 ] | |
|
Closing this issue as fixed, I opened follow-up issue related to high memory consumption | |
| Comment by Nick Judson [ 27/Jul/15 ] | |
|
Testing with a 1GB cache was slightly slower than 4, with the snappy run completing in 10h:15m. I noticed it was using 3GB of memory, so some investigation may be required for that. | |
| Comment by Eitan Klein [ 27/Jul/15 ] | |
|
Thank you Nick for deep dive into this subject, and I personally enjoyed to cooperate w/ you on this!
In addition to the running w/ lower cache, it will be interesting to know the impact of TTL indexes w/ your scenario Thanks again, Eitan | |
| Comment by Nick Judson [ 27/Jul/15 ] | |
|
3.0.0, zlib 4GB cache: 10h:11m (but actual memory usage much higher). Overall everything is looking good. I didn't expect there to be such a (negative) impact on performance when giving more memory to the WT cache, but that is most likely just something to document. 3,0,0 was slightly faster for zlib, but the performance graphs do look much nicer for 3.1.6. I think this ticket can probably be closed. Now I'll test with an even lower cache. | |
| Comment by Nick Judson [ 25/Jul/15 ] | |
|
Eitan et. al, I reran the zlib test on 3.0.0 just to re-confirm my suspicion that it has slowed in 3.1.6. I've attached the results above confirming that it took 10h:11m with a file size of 100GB even. I'm going to re-run this exact test with 3.1.6 for comparison. mongod --dbpath=e:\mongo --port=27200 --storageEngine wiredTiger --wiredTigerCacheSizeGB=4 --wiredTigerCollectionBlockCompressor=zlib --wiredTigerJournalCompressor=zlib | |
| Comment by Eitan Klein [ 23/Jul/15 ] | |
|
Hey Nick, Thanks again for the information, very interesting finding Few questions
Thanks again, Eitan | |
| Comment by Nick Judson [ 23/Jul/15 ] | |
|
Eitan, my zlib test didn't complete after 16+ hours so either there is a regression there or the performance counters are reasonably heavy. I'll have to retest without the perf counters tonight. Strangely, the file size is 152GB for only 240M docs. My original tests had the full 300M docs at only 100GB...so perhaps something has changed there? Memory footprint was a little larger at 9.6GB (than the snappy run). I'll let you know tomorrow the results without the perf counters. | |
| Comment by Nick Judson [ 23/Jul/15 ] | |
|
Perf counter zlib 3.1.6 | |
| Comment by Eitan Klein [ 22/Jul/15 ] | |
|
Thanks Nick for you quick engagement on this, I'm very happy you see the improvement On the zlib run can you turn on the performance counter data and upload and share Appreciate your engagement , Eitan | |
| Comment by Nick Judson [ 22/Jul/15 ] | |
|
I did a 'snappy' run last night using 3.1.6 (chart uploaded). The performance looks much more consistent now and the test completes in 10h (although there were a few other things happening on the box so probably a hair quicker in reality). The folder size is 180GB, which is slightly smaller than previous runs. I left the memory flag empty and on my 16GB box mongod is using 8.7 GB of ram, which seems perfectly acceptable. Next I'll run the same test with zlib and upload the results. | |
| Comment by Nick Judson [ 21/Jul/15 ] | |
|
Sure thing, I'll pull it into our experimental branch and let you know. | |
| Comment by Eitan Klein [ 21/Jul/15 ] | |
|
Hey Nick, I hope you are doing well, we just released 3.1.6 that have many performance improvements. I'm looking forward to hear back from you, | |
| Comment by Ramon Fernandez Marina [ 17/Jun/15 ] | |
|
After further consideration I'm going to reopen this ticket to make sure we come back to it after some/all the other performance-related issues that Eitan is tracking are addressed. | |
| Comment by Nick Judson [ 15/Jun/15 ] | |
|
Fine by me. I left some debugging info with Eitan last month. To summarize what I've learned thus far: For version 3.0.0, zlib unexpectedly outperforms snappy for my workload. This may be due to some efficiencies in zlib which are not available in snappy. Testing newer builds provided by Eitan reversed that trend. Snappy became faster and zlib became slower. Issues were observed which appeared to be similar to other tickets raised by Eitan, and therefore this one can be closed pending testing (once the other tickets are addressed). Interestingly, testing with the new memory allocator (for wt only) didn't provide the big performance jump expected. I think some of the other issues need to be tackled before throwing this workload at mongodb. | |
| Comment by Ramon Fernandez Marina [ 15/Jun/15 ] | |
|
nickj, this ticket doesn't contain enough information to point to a bug in the server, so I'm going to close the ticket. I understand that you've been in touch with eitan.klein, so if you have further results or want to test other builds feel free to reach out to Eitan or post comments/questions here. Regards, | |
| Comment by Ramon Fernandez Marina [ 29/Apr/15 ] | |
|
nickj, thanks for all the testing you've done so far. If I understand correctly there are multiple variables you're considering such as operating system, number of documents, underlying hardware... Other aspects that may be interesting to test are using WT stand-alone, or using a different allocator on linux. It would also be useful to know if you're using stock builds or if you're doing your own builds. And it would be of great help if you could share with us the code you're using, so we can speed up the investigation. You can upload the code securely and privately (i.e.: only accessible to MongoDB staff for debugging this ticket) via scp:
Where <filename> is a zip/tgz archive of the files to upload; when prompted for a password just hit enter. Do you think you can share your testing code with us to help us reproduce the behavior you're seeing? Thanks, | |
| Comment by Nick Judson [ 29/Apr/15 ] | |
|
Same test as above but zlib. Test takes 11h:52m. So snappy is, in fact, faster, albeit by a very small amount. I guess the conclusion is that mongodb using zlib degrades less than snappy in this scenario. | |
| Comment by Nick Judson [ 28/Apr/15 ] | |
|
It would appear that the length of the test is a significant factor in the results. For example, (using Snappy one the same box), 125M documents takes 3h:46m. 300M takes 11h:22m. Extrapolating from the short test, the long one should take about 9 hours. Something appears to change at around the 3 hour mark and performance becomes choppy. This may be where zlib has a chance to catch up. I will test that next. | |
| Comment by Nick Judson [ 23/Apr/15 ] | |
|
Two more test with 125M docs: zlib and snappy. Snappy is faster in this test. Last test will be the full 300M docs with both compression types. If those act as expected then the issue was either in 3.0.1 or the beta C# driver or my app itself. | |
| Comment by Nick Judson [ 21/Apr/15 ] | |
|
As above, except zlib. Test completes in 3:54:02. For both linux and wiindows in the app server deployment, snappy is faster than zlib (which is the expected result). This is a cut down test of only 125M docs, so either the test size or the deployment configuration (app server vs stand-alone) appears to have an effect. | |
| Comment by Nick Judson [ 20/Apr/15 ] | |
|
Using w2012R2 I ran the 125M message test, which completed in 2:56:50 - significantly faster than on ubuntu. There is a good chance I didn't have something set perfectly on ubuntu however. Next test will be zlib on windows (via app server). | |
| Comment by Nick Judson [ 17/Apr/15 ] | |
|
I re-ran the last test using zlib in place of snappy (on ubuntu). Test completed in 3:47:28, making snappy 8% faster than zlib in this test.If the full 300M docs was run would zlib would pull ahead? From the graphs, performance looks stable for both compression types. | |
| Comment by Nick Judson [ 16/Apr/15 ] | |
|
I ran a quick test using snappy on ubuntu 14.04, with only 125M messages. The test completed in 3.5 hours (3:30:30). | |
| Comment by Nick Judson [ 14/Apr/15 ] | |
|
A test was run with the zlib_noraw flag for wiredTiger. The performance degraded significantly and the test was killed after 14.5 hours. The db size was 135GB. Per @Mark Benvenuto, this may indicate why zlib may be comparable to snappy in certain circumstances (throughput-wise). | |
| Comment by Nick Judson [ 13/Apr/15 ] | |
|
I've run two of the four tests now: snappy vs zlib on windows 8.1. Snappy completes in 11:22:30 and zlib completes in 11:01:04. You can see from the attached charts that snappy has a much higher variance than zlib, although this may be unrelated. Snappy creates a larger database (186 vs 100 GB) but this test barely scratches the IO, so that shouldn't be part of the equation. In my own personal tests of snappy vs zlib (outside of wiredTiger) on the same data, snappy was roughly 10x. I'm in need of a SSD for testing on Linux, and when I pick one up I'll run the same test using the current machine as the application server and wiredTiger on ubuntu 14.04. | |
| Comment by Nick Judson [ 10/Apr/15 ] | |
|
Assuming the remaining tests warrant further investigation, I will provide an exe for repro. | |
| Comment by Nick Judson [ 10/Apr/15 ] | |
|
A quick overview about the test: I am taking 50 streams of text and 50 threads are inserting that text into 50 collections within a single database. Data is read from those 50 collections, lightly manipulated, and then a light update operation is done (no increase in field size) along with an insert into another collection. The read and update/insert cycle happens on a separate thread. So, we have ([1 x thread insert] + [1 x thread read, update, insert]) x 50. Each of the insert threads is inserting 6M docs, giving a grand total of 300m docs. All inserts and updates are done in bulk. I'm running some new baseline tests with version 3.0.2 and the 2.0.0 release C# driver. Using WiredTiger with both collection and journal compression set to zlib/snappy and a max cache size of 4GB. The IO is PCI attached SSD, so zlib should have no advantage writing fewer bytes. The zlib graph shows running the 300m doc test took 11 hours to complete. Speeds slow down initially but settle into a nice steady-state pattern. The test finishes in 11 hours and the database is 100GB. The next test will be using Snappy instead of zlib. I there are already several tickets for the memory usage, but you can see that with a max cache set to 4GB, the usage at the end of the test is between 6 and 7GB (and if you set the max to 6, it ends up in the 8 to 9 range). |