[SERVER-11117] Tagged Secondary Preffered not working as expected - Reads always happening at Primary Created: 10/Oct/13 Updated: 07/Nov/13 Resolved: 01/Nov/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Sharding |
| Affects Version/s: | 2.4.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Vinod Kumar | Assignee: | A. Jesse Jiryu Davis |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | query_triage | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Running on RHEL , Java driver 2.11.2 |
||
| Issue Links: |
|
||||||||||||||||
| Operating System: | Linux | ||||||||||||||||
| Steps To Reproduce: | Our test env is a mongos router with 2 shards - 3 mongod's and one arbiter per shard and Query generated from Java App . using 2.11.2 driver using tagged secondary preffered mode |
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
During testing we found that setting read Preference to "secondaryPreferred" always fetched records from primary even if the tag was invalid . But the online documentation suggests an error should be thrown " http://docs.mongodb.org/manual/reference/read-preference/#secondaryPreferred " . This leads me to believe Mongos/Java Driver is not respecting tags or behaving otherwise and always reading from primary . Since our actual deployment is large and geographically we need multiple secondaries |
| Comments |
| Comment by A. Jesse Jiryu Davis [ 01/Nov/13 ] | ||
|
Vinod, feel free to reopen this issue if you find a way to reproduce it. | ||
| Comment by A. Jesse Jiryu Davis [ 30/Oct/13 ] | ||
|
Vinod, are you currently able or unable to reproduce this problem? If you're unable, then If you are able to reproduce this issue, could you please explain to me the steps you take to reproduce it, with code, and show me the log messages from mongos during your test? | ||
| Comment by Vinod Kumar [ 24/Oct/13 ] | ||
|
Just ran a query with both secondary & secondaryPreferred modes with both shard and non sharded keys .No such error was reported in the logs . If however there was null in the tag set , I guess the proper behavior should be it should not consider that as a tag and work as usual . Any idea on | ||
| Comment by A. Jesse Jiryu Davis [ 23/Oct/13 ] | ||
|
Vinod, read preferences like secondaryPreferred apply at the shard level. When you tell mongos that you prefer to read from secondaries, it applies that preference to each replica set. So, if one replica set has no secondaries, then mongos will read from the primary, whereas if another replica set has a secondary, mongos reads from the secondary. I'm very curious about the "User Assertion: 16357:Tags should be a BSON object" message. If it's present in your log, then we'll know for certain the cause of the misbehavior you're seeing. | ||
| Comment by Vinod Kumar [ 23/Oct/13 ] | ||
|
OK will check and get back .. This query I got was from an older log file . hence need to ask the team to do a fresh run and get the output . A quick confirmation question for you . Is the secondaryPreffered option at a shard level or at the mongos level ? For e.g. If I were to use secondaryPreffered and one of the secondaries in one of the shards were to go down , would all the shard queries be routed to their primary nodes or only that shard's query would be routed to its primary . I just found a similar issue here | ||
| Comment by A. Jesse Jiryu Davis [ 22/Oct/13 ] | ||
|
Can you confirm you're still seeing log messages like this one from mongos?:
In other words, the list of tags ends in null? In that case mongos should also log:
Is that message in your mongos log? | ||
| Comment by A. Jesse Jiryu Davis [ 22/Oct/13 ] | ||
|
Vinod, secondaryPreferred routes queries to secondaries, if any are available which match your tags. That's why I want to see some of your misbehaving code, to help you diagnose why your particular usage of secondaryPreferred is not giving you the results you want. Can I at least see your Java code where you configure the tags to use? Or, can you reproduce the problem using the mongo shell, and show me the shell code you used to reproduce this issue? To verify for certain which member is used for a query, you can either watch the members' logs, or turn on profiling on each member, following the technique I outline here: http://emptysqua.re/blog/real-time-profiling-a-mongodb-cluster/ | ||
| Comment by Vinod Kumar [ 22/Oct/13 ] | ||
|
Hi, Is there a sure shot way of verifying the behaviour of the query being routed to secondary (while using secondary preferred) ? Coming to our code it has lot of frame work code around it ,hence cant post the whole of it . Basically in Java we set all options including secondary preferred in MongoClientOptions and use Parameters used readPreference - > secondary | ||
| Comment by A. Jesse Jiryu Davis [ 18/Oct/13 ] | ||
|
To avoid any further confusion, would it be possible for you to paste some actual unmodified code, which reproduces this problem, where I can see how you specify your read preference and tags? Either Java code or Mongo shell code is fine. Thanks. | ||
| Comment by Vinod Kumar [ 18/Oct/13 ] | ||
|
Actually I had pruned some of the prod related terms with equiavalent terms , hence you see a slight mismatch in what I copy/pasted . Sorry about that .. In our case validTag = b .Also we are passing only one tag ,Maybe thats the reason why you notice an empty {} | ||
| Comment by A. Jesse Jiryu Davis [ 17/Oct/13 ] | ||
|
The driver appears to be passing null at the end of the tag sets instead of {}, and mongos is misreporting the error. | ||
| Comment by A. Jesse Jiryu Davis [ 17/Oct/13 ] | ||
|
Thanks. I expect that if you replace {"locations": "validTag"}with a tag like {"locations": "b"}and use mode secondaryPreferred, you'll read from the secondary tagged with "locations": "b", that is, you'll read from "psenv1.ugi:27010". Since "validTag" doesn't actually match any tags in your system, mongos falls back to reading from primary with mode secondaryPreferred. Alternatively, if you'd like to fall back to reading from any secondary in the case that no secondary matches your tag, include "{}" as the final tag set:
I notice in the mongos log you showed me that the final tag isn't "{}", it's "null". Can I see the Java code with which you specify your read preference and tags? | ||
| Comment by Vinod Kumar [ 17/Oct/13 ] | ||
|
Current output of rs.conf and rs.status }, }, , } ----------------------------------------------------------------------------- , , , { "_id" : 3, "name" : "psenv4.ugi:27010", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 162458, "optime" : Timestamp(1382000647, 1), "optimeDate" : ISODate("2013-10-17T09:04:07Z"), "lastHeartbeat" : ISODate("2013-10-17T09:04:09Z"), "lastHeartbeatRecv" : ISODate("2013-10-17T09:04:09Z"), "pingMs" : 5, "syncingTo" : "portals1.ugi:27010" } ], | ||
| Comment by A. Jesse Jiryu Davis [ 15/Oct/13 ] | ||
|
Could I see the output of rs.conf() and rs.status() when connected to the primary of the relevant replica set, please? | ||
| Comment by Vinod Kumar [ 15/Oct/13 ] | ||
|
Here is some logging information we captured from Mongos ( connected to 2 shards with 3 RS members each) a few weeks back on our QA env . Basically our code generates query using secondaryPreffered with a tag as can be seen below . Wed Sep 11 10:36:47.369 [conn648] shard query: u_db.enr { $query: { _id: { $in: [ "someKey" ] }}, $readPreference: { mode: "secondaryPreferred", tags: [ { locations: "validTag" }, null ] } We arrived at the conclusion that the bunch of queries were not firing to primary and not the secondary by looking at the mongod log on one of the replica set members which was 'primary ' at that point in time. Is there a sure shot way of verifying the behaviour or should we stick to secondary instead of secondary preffered ? Cheers | ||
| Comment by A. Jesse Jiryu Davis [ 11/Oct/13 ] | ||
|
vinod.k secondaryPreferred reads should go to secondaries if any are available, and if at least one matches your tags.
| ||
| Comment by Vinod Kumar [ 11/Oct/13 ] | ||
|
We also found that reads for some reason in secondaryPreffered mode always goes to primary . But it works well with secondary . | ||
| Comment by A. Jesse Jiryu Davis [ 10/Oct/13 ] | ||
|
Sorry, this is a problem with the documentation. The behavior you observe is correct: if no secondaries match your tags, mongos and the drivers should ignore tags and read from the primary. I've submitted a patch to the documentation. |