[SERVER-21838] Replica set with mixed compression types slower than expected Created: 10/Dec/15 Updated: 20/Jun/16 Resolved: 20/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Nick Judson | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Operating System: | ALL | ||||
| Steps To Reproduce: | Create a replica set with the primary using zlib and the secondary using snappy. Check the CPU usage and the performance characteristics vs zlib/zlib and snappy/snappy. zlib/snappy appears to be roughly 25% slower than zlib/zlib, which the CPU only running around 50% with zlib/snappy configured. [Nick J workload] |
||||
| Participants: | |||||
| Description |
|
I'm playing with replica sets on 3.2. I have the following topology: 1 x i3770 with SSD [Primary] .NET application using the C# driver (2.2). Scenario 1 - No replica set: My application running on a separate box connects to the primary (with no replica set configured) and has a total throughput of X. Scenario 2 - 1 node replica set (on the primary): Same two boxes as above, but mongod on the primary is started with --replSet. Total throughput is X - 15%. This makes sense as there is extra CPU and disk IO required. Scenario 3 - 2 node replica with arbiter: This time I've configured a standard replica set. Throughput drops to X - 55%. I can see the NUC (which is very weak) is CPU-bound, and the other two boxes are barely breaking a sweat. My understanding was that replication was asynchronous and that the replication from primary to secondary would not/should not slow down writing to the primary (at least not by such a large amount). As best I can tell I don't have the write concern set to majority (unless that is the default for a cluster). I noticed that on my primary I was using zlib compression for both journal and collection (primary has a smaller SSD) and was using snappy for the replica. I tried using snappy on both and performance jumped up more than expected, and CPU on the primary popped up to 100%. I also tried zlib on both primary and secondary, and this showed better performance that mixed. |
| Comments |
| Comment by Ramon Fernandez Marina [ 20/Jun/16 ] |
|
nick@innsenroute.com, I'm going to close this ticket based on your last comment:
We've been working to improve the write performance in primary nodes as part of If you want to do any further testing with 3.3.x and find unexpected behavior feel free to open a new SERVER ticket. Thanks, |
| Comment by Nick Judson [ 10/Feb/16 ] |
|
I think the confusion is that my initial test involved a 3-node replica set where one machine had limited disk space and so I was using zlib (whereas the primary was using snappy). When I compared the speeds of a local-only replica-set to this setup, the speeds were way slower. My expectation was that the secondary would fall behind, but that the primary would not significantly slow down (I'm not using w:majority). My initial conclusion was that there was a problem using different compressions across nodes. But then I switched everything to the same compression and I saw the same behavior: Local-only replica-set primary speed = X. Add secondary node, primary speed = Y. I understand there will be some performance hit with the proper replica-set, but not a 50% reduction on the primary. My setup was as simple as it gets. Windows x 3 (win 10 primary, 2k12 secondary, 2k12 arbiter). Couldn't get any non-local replica-set to run without a major perf hit. |
| Comment by Ramon Fernandez Marina [ 10/Feb/16 ] |
|
Hi nick@innsenroute.com, are you still pursuing these benchmarks? After re-reading the ticket I found it difficult to get a clear idea of the setup, and as others mentioned above there are a few open questions about your setup that make it hard to attempt to reproduce your observations here. The diagnostics data that 3.2 collects is also missing, so it's hard to start looking for clues. Please let us know if you still want to continue exploring this ticket or if you're still seeing performance anomalies on mixed compression replica sets. Thanks, |
| Comment by Nick Judson [ 28/Dec/15 ] |
|
(attachment) Quick run with single node RS. Using the primary from the previous run, start with a fresh mongod and create a RS (and initiate it). Run the load generator. This leaves out the secondary and the arbiter, but everything else is the same. |
| Comment by Nick Judson [ 28/Dec/15 ] |
|
(Attachments added) Short run with 3 node RS (2 nodes, 1 arb): Start with 3 fresh mongod instances, create RS, run load generator. Performance much slower than single node RS (next attachment) |
| Comment by Daniel Pasette (Inactive) [ 28/Dec/15 ] |
|
Hi Nick, Also, in v3.2 we have enhanced diagnostics capture on by default. Documentation here. If you can attach the contents of the diagnostics directory and log files for each replica set node, we'll be able to get a much better idea of what is happening. Thanks! |
| Comment by Nick Judson [ 28/Dec/15 ] |
|
I don't know if this is |
| Comment by Nick Judson [ 28/Dec/15 ] |
This was my expectation as well. I was just getting up to speed with replica sets on my usual queue implementation and didn't understand what I was seeing. I have not explicitly set w:majority - are you saying this is now an implicit default in 3.2? (I don't think you are). I'm using the latest C# driver. |
| Comment by Daniel Pasette (Inactive) [ 28/Dec/15 ] |
|
Ordinarily I would expect the slower machine to simply fall behind the primary but not cause the primary to get slower. Which workload is this? If you are using a w:majority workload with replset protocol version 1 (the default for new replsets in v3.2), you might see some slowdowns though – |
| Comment by Nick Judson [ 24/Dec/15 ] |
|
I just re-tested this and am still seeing the following behavior: Create local-only replica set. Performance = X. My understanding was that the slow node should not slow down the primary (much). Not really sure what else to test, other than moving the slow node onto the fast network when I get back to work. |
| Comment by Nick Judson [ 10/Dec/15 ] |
|
Hi Ramon, I think it would be helpful to have a shared screen session as I can't really make heads or tails of what I'm seeing. Do you want to have someone PM me and we can schedule something? N. |
| Comment by Ramon Fernandez Marina [ 10/Dec/15 ] |
|
I've put this ticket in the "debugging with submitter" category while you continue to run your experiments and gather data. Please note that if you're running your tests on Windows you may be affected by Cheers, |
| Comment by Nick Judson [ 10/Dec/15 ] |
|
Now I'm not sure if I've stumbled upon a larger issue. My snappy/snappy run was slow, so I started from scratch with a single node replica set which was fast. I added a second node and performance fell off a cliff. I'll try this same experiment with a zlib/zlib cluster tomorrow. Summary so far: |
| Comment by Nick Judson [ 10/Dec/15 ] |
|
Just running a snappy/snappy test and I'm seeing the same 50% cpu usage (both primary and secondary) and slow speeds. I'm not sure where the bottleneck is here - lots of IO and network bandwidth. I'll attach some info from the snappy/snappy run. |