[SERVER-6604] slaveOk reads are going to primary Created: 26/Jul/12 Updated: 15/Aug/12 Resolved: 26/Jul/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.2.0-rc0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Pierre Dane | Assignee: | Robert Stam |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows 64 with c# driver |
||
| Operating System: | Windows |
| Participants: |
| Description |
|
After upgrading our 2 secondaries to 2.2 rc0 with master still at 2.0.6, driver starts reading from master as well as secondaries. Switching the secondaries back to 2.0.6 resolves the problem. This could be specific to the c# driver, or might be resolved once the master is also upgraded (waiting for production release before upgrading master). |
| Comments |
| Comment by Robert Stam [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Pierre, I'm marking this issue as Resolved for now. Please reopen or submit a new ticket if you would like to pursue further in the future. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
ok Thanks for looking into this Robert - I'll wait for the next release and try to replicate this problem again. If I can I will send you full historical logs and mongostat for all three nodes. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Robert Stam [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
I tried to reproduce this but am unable to. My test program which does slaveOk reads is evenly distributing the reads to the secondaries. Here's my mongostat output while running the test:
| ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
The reads from the secondaries to the primaries during resync would be getmore commands though wouldn't they? | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
localhost:3002 *6 1411 *73 *0 0 22|0 0 24.1g 39g 19.5g 167 0 0 0|0 1|0 110k 814k 7325 tagstore SEC 12:39:40 | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
I only resynced one a time, and they were both fully synced when I was seeing the issue. I was seeing reads off all three servers, and the reads were not symmetric - the load was swinging wildly | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Robert Stam [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
A secondary that is doing a full resync can't be used for reads until it is caught up. If both of your secondaries are doing a full resync at the same time neither one of them is available for reads, so the driver will send all slaveOk reads to the primary until at least one of the secondaries is caught up. This would definitely result in a high read load on your primary, because not only are all your application reads going to the primary, but both secondaries are doing lots of reads as well as part of the full resync process. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
We are using v1.5 The following data was captured when upgrading just one secondary to 2.2, with a full resync. i.e. deleted all data and started the secondary. The read load on the master is not high in the mongostat below, but when both secondaries are upgraded the load goes up into the thousands. Feel free to check MMC - you can see the read spike on the primary (mongo88) STARTUP2> rs.status() , , , , localhost:3002 *0 *0 *0 *0 0 20|0 0 11.2g 18g 732m 3679 15.1 0 1|0 2|0 1k 9k 6794 tagstore UNK 14:39:27 | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Robert Stam [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Normally the C# driver only sends slaveOk reads to the primary if no secondary is available. The C# driver uses the ismaster command to query the state of each replica set member and looks for { secondary : true }in the response to determine whether a replica set member is currently a secondary or not, so another useful piece of information would be run the following command in the mongo shell against each secondary that is not receiving the queries you expect:
| ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Robert Stam [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Can you also please report which version of the C# driver you are currently using? Thanks. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Pierre Dane [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Hey Scott, Connecting directly to the replica set. I've reverted to 2.0.6 but I'll upgrade again and post rs.status – anything else you want me to get while I upgrade for a short period? | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Scott Hernandez (Inactive) [ 26/Jul/12 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Are you using sharding with a mongos instance, or is your .net client directly connecting to the replica set? What does rs.status() show? |