[SERVER-25842] Secondary started accepting queries before syncing(replSet still syncing, not yet to minValid optime) Created: 29/Aug/16 Updated: 30/Aug/16 Resolved: 30/Aug/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying, Replication |
| Affects Version/s: | 2.6.11 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Manoj Vivek | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: |
|
||||||||
| Participants: | |||||||||
| Description |
|
I have brought up a new replica from EBS snapshot(2 days old) of an existing secondary and then added it to the replicaset using rs.add(). But once added it started serving queries immediately even before fully syncing with the primary.
Logs:
Made a workaround by removing and making a hidden secondary till it got synced. |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 30/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Manoj, I think what you're seeing is "Worked as Designed." The read preference allowing clients to read from secondary nodes does not check how far behind it is from the primary node. Coming in the next version of MongoDB (v3.4) users will be able to set a read preference specifying the maximum staleness a client will accept before falling back to querying the primary. You can see the details here: In the meantime, adding the node as hidden while it catches up or changing read preference to primary only is the best workaround to this situation. Thanks | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Manoj Vivek [ 30/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
ramon.fernandez I have raised this in the mongodb-user group here https://groups.google.com/forum/#!topic/mongodb-user/jC8_t6jeojU To reply to your comment, even after the state got changed to SECONDARY rs.status() showed that this new member is more than 10 hours behind the primary and catching up(but serving the queries). Anyways this is something I have witnessed but didn't gather enough data during that time to back it up now. Also reproducing it now involves quite a bit of time and effort. I will get back if I get a chance to redo this again. Thanks for your time. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 30/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
vivek_jonam, I don't see any queries in the secondary before it transitions to SECONDARY. Note that the node can accept connections, but it will not serve any data. One thing you can try is to configure your secondary to log all operations. If you see any queries please send the logs, but for not I'm closing this ticket as I don't see evidence of a bug. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources. Regards, | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Manoj Vivek [ 30/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
ramon.fernandez Please ignore the logs in the description and consider the following logs. Apologize for the different timezones in primary and the new secondary logs.
Logs in new secondary:
Let me know if you need any thing else. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 29/Aug/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
vivek_jonam, if I'm not mistaken, what the log snippet above shows is that the replica set is still syncing at 5:32:42, but it finishes syncing at 5:32:51 and then transitions to SECONDARY, at which point is ready to accept queries. Perhaps if you upload the full logs from this secondary from startup, as well as the corresponding logs on the primary for the same time period, we may be able to see more. |