[CSHARP-1053] Major performance degradation from 1.6.1 to 1.9.2 Created: 28/Aug/14  Updated: 04/Apr/15  Resolved: 04/Apr/15

Status: Closed
Project: C# Driver
Component/s: Performance
Affects Version/s: 1.9, 1.9.1, 1.9.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Anthony Sutton Assignee: Unassigned
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File MongoDriverPerf.1.6.1140829.vsp     File MongoDriverPerf.1.9.2140829.vsps     PNG File post-deploy-stats.png    

 Description   

I have thrown together a performance demonstration for this ticket here: https://github.com/squirrelsama/mongo-csharp-driver-performance-issue

The short of it: When my team upgraded the driver from 1.6.1 to 1.9.2, we were hit with serious performance issues and had to roll back. We upgraded specifically because we need the ReadEncoding setting support introduced in 1.8 and will use 1.8 until this issue gets addressed. More details in the github link, including a runnable perf script that takes ~10 minutes.

I believe this issue is major because with the 1.6.1 driver, mongo will perform the test in just under 30 seconds; but in 1.9.2, that time doubles to 60 seconds. This likely implies that at minimum, half the time spent trying to interact with mongodb is spent in the driver's code, not waiting on mongodb's network.

If you want more information, I'd be more than happy to oblige.



 Comments   
Comment by Craig Wilson [ 04/Sep/14 ]

Hi Anthony,

Thanks for the report. I'm concerned a little that even 1.8.3 is worse than 1.6. However, 1.9 is definitely going to be slower in certain cases. The server now prefers the use of write commands, and there is a lot of logic that goes into splitting batches correctly and us needing to decide whether to use the old opcodes or the new write commands. Using 1.6 will prevent you from using newer versions of the server when they remove the opcodes (yet to be determined). In addition, if you are using server versions 2.6 in a sharded environment, there is no getting around this overhead at all as mongos does the same thing we are doing.

Long and short... we will work on improving performance and I apologize for the drift. Part of it is unavoidable, but we'll see what we can do to amp it back up.

Craig

Comment by Anthony Sutton [ 04/Sep/14 ]

Attached are some mongo stats for deploying from 1.6.1 to 1.8.3.

For now, my plan is to fork 1.6.1 to add the ReadEncoding feature we need, and defer upgrading until these performance issues are resolved.

Comment by Anthony Sutton [ 29/Aug/14 ]

FYI, just to clarify: We did observe the initial performance issue that required rollback with the client running on Windows, mongod running on Linux. Only my profiling was done with the client running on Linux/mono

Comment by Jose Luis Pedrosa [ 29/Aug/14 ]

Attached the profiling from 1.6.

Comment by Jose Luis Pedrosa [ 29/Aug/14 ]

profiling from visual studio running the test of 1.9.2

Comment by Jose Luis Pedrosa [ 29/Aug/14 ]

Hi,

I've run the test in windows 7 64 bits, Release version. this are the results:

from VS
1.6.1.4678 took 96065ms
1.9.2.235 took 122188ms

From cmdline
1.6.1.4678 took 79630ms
1.9.2.235 took 98287ms

Indeed just by browsing the task manager, I can see that in version 1.6 the mongod proc uses more CPU than the load test generator, while on 1.9 it's 50%-50%. The diferences are not as high as the one that Anthony had with mono.

Looking at the performance profiling, aparently, Insert operation is the one that has become heavier.

BR

Comment by Anthony Sutton [ 28/Aug/14 ]

Typo at the end: "not waiting on mongodb OR the network."

Generated at Wed Feb 07 21:38:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.