[CSHARP-574] Secondary Reads shouldn't use a Primary if a Secondary is available Created: 19/Sep/12  Updated: 06/Sep/16  Resolved: 19/Oct/12

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 1.6
Fix Version/s: 1.8

Type: Bug Priority: Minor - P4
Reporter: Michael Ballou Assignee: Robert Stam
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 2008 R2



 Description   

UPDATE:
The code base currently takes primary ping times into consideration when using a secondary preferred read preference, which causes the below issue. In addition, we plan to make the acceptable latency (currently at 15ms) configurable.

ORIGINAL:

I have 3 Amazon EC2 instances setup. Machine A runs Windows 2008 R2 using the 1.6 C# Driver. The Mongo systems are running on Ubuntu 12.
Machine A: web site (US-West zone)
Machine B: Mongo 2.2.0 primary (US-West zone)
Machine C: Mongo 2.2.0 secondary (US-East zone)

The only way I'm able to get queries to connect to the secondary, is if I set the readPreference to secondary. Setting to secondaryPreferred always picks the primary so my primary gets overloaded and the secondary is never used.

The bug seems to come from MultipleInstanceMongoServerProxy.Connect(). When ConnectInstance is called, it starts connections to both servers. The primary connects right away, but since this function only sleeps for 20 milliseconds, the secondary (in another amazon zone) doesn't get a chance to finish completing by the time the loops runs again. So ChooseServerInstance gets called with only the primary connected, so the secondary is not in the list by the time ReadPreference.ChooseServerInstanec is called.

I saw this behavior happen in the debugger.



 Comments   
Comment by Githook User [ 06/Sep/16 ]

Author:

{'name': 'Craig Wilson', 'username': 'craiggwilson', 'email': 'craiggwilson@gmail.com'}

Message: CSHARP-574: minor changes.
Branch: master
https://github.com/10gen/sqlproxy/commit/baca05fab070c41728eeb68364a8090e290c3f76

Comment by auto [ 06/Nov/12 ]

Author:

{u'date': u'2012-10-19T16:31:48Z', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: SecondaryAcceptableLatency was missing from MongoUrl.

Conflicts:
MongoDB.DriverUnitTests/MongoUrlTests.cs
Branch: x2.0
https://github.com/mongodb/mongo-csharp-driver/commit/ea5cf57e3ec21dcfd6c71efcb2e4320f0260b5d6

Comment by auto [ 19/Oct/12 ]

Author:

{u'date': u'2012-10-19T09:31:48-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: SecondaryAcceptableLatency was missing from MongoUrl.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/e577aac7f59548817a6e51c22cd79022fefee319

Comment by Robert Stam [ 19/Oct/12 ]

Still need to add SecondaryAcceptableLatency to MongoUrl.

Comment by auto [ 26/Sep/12 ]

Author:

{u'date': u'2012-09-24T12:50:36-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: special case single matching instance and put lock around call to Random.Next.
Branch: x2.0
https://github.com/mongodb/mongo-csharp-driver/commit/061a30bcab4dad03e62bf55d8ff04a92e0452dd7

Comment by auto [ 26/Sep/12 ]

Author:

{u'date': u'2012-09-20T07:48:56-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: refactor so callers of GetMatchingInstances decide which instances should be matched against.
Branch: x2.0
https://github.com/mongodb/mongo-csharp-driver/commit/efbacdfb92f2825e3649991578317018de3ff766

Comment by auto [ 26/Sep/12 ]

Author:

{u'date': u'2012-09-20T05:47:27-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: reduce possibility of lock contention and move matching logic out of ConnectedInstanceCollection and into ReplicaSetMongoServerProxy.
Branch: x2.0
https://github.com/mongodb/mongo-csharp-driver/commit/a33ae01666c331d140565d5d05ae0936bc8ed2e1

Comment by auto [ 26/Sep/12 ]

Author:

{u'date': u'2012-09-19T22:47:38-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: refactor the ReadPreference implementation and fix some issues.
Branch: x2.0
https://github.com/mongodb/mongo-csharp-driver/commit/74d21a5f1f6231d7bdabd7f7b50c2b89b77a9698

Comment by auto [ 25/Sep/12 ]

Author:

{u'date': u'2012-09-24T12:50:36-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: special case single matching instance and put lock around call to Random.Next.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/34bbb5b97739a4d2fdcdc56f8b56627274ced2d8

Comment by auto [ 25/Sep/12 ]

Author:

{u'date': u'2012-09-20T07:48:56-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: refactor so callers of GetMatchingInstances decide which instances should be matched against.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/dc4ece4c3706111739024da8ec9b11a7c89996fc

Comment by auto [ 25/Sep/12 ]

Author:

{u'date': u'2012-09-20T05:47:27-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: reduce possibility of lock contention and move matching logic out of ConnectedInstanceCollection and into ReplicaSetMongoServerProxy.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/3f74e5cc0a5bd6df7c86415805c1a3fea74e0ab7

Comment by auto [ 25/Sep/12 ]

Author:

{u'date': u'2012-09-19T22:47:38-07:00', u'email': u'robert@10gen.com', u'name': u'rstam'}

Message: CSHARP-574: refactor the ReadPreference implementation and fix some issues.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/8549cb72f04f511aa470618cc5510ad2a00b3fa5

Comment by Craig Wilson [ 20/Sep/12 ]

Mike, this is actually a bug in our system. When you use secondary preferred, we should actually be ignoring the primary if a secondary is available. I retitled and edited the description of the Jira.

Keep in mind in this case that all your reads are going to go all the way across the country to get information.

Our suggestion in the immediate case is to make your US-East secondary the hidden member and your hidden US-West secondary live. I would actually suggest this action regardless because of the above mentioned latency of always going across country.

Comment by Michael Ballou [ 19/Sep/12 ]

Thanks Craig.
I do have a 3rd member in the West, but it's a smaller instance and its configured as hidden with 0 priority.

You are correct on the ping times. The West coast mongo pings are less than 1ms. The East coast times are 84ms. Given that, it is outside the 15 millisecond window and is never chosen.

I don't see a huge demand for this behavior. It's really only useful when spreading data to multiple data-centers for redundancy, but the proper way would be for me to have multiple replicas in each zone. I'm in the process of setting up more replicas in the west as a result.

Maybe just a comment in the release notes/readme would suffice so someone else can benefit from that knowledge. Or some kind of driver logging that can show why a particular server is chosen.

Comment by Craig Wilson [ 19/Sep/12 ]

That is true about the initial connect where we will return as soon as we possibly can to get you up and running. However, connect is only used once. The rest of the time ChooseServerInstance is used...

The driver maintains a list of average ping times for each instance it is connected to. The way all the drivers implement this particular algorithm is the same. In the case where multiple instances are possible (secondary, and secondary preferred). Basically, we find all the servers matching both the read preference and the tags if any are specified, and then randomly choose an instance from those within 15 milliseconds of lowest ping time. You can see this behavior here: https://github.com/mongodb/mongo-csharp-driver/blob/master/Driver/Core/ReadPreference.cs#L399.

Do you only have a 2 member replica set? Given your setup, I am not surprised that your secondary never gets hit as the request has to travel all the way across the country. We could make this functionality pluggable, but that doesn't exist at this point in time. What are your thoughts?

Generated at Wed Feb 07 21:37:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.