[SERVER-53474] Cannot read from another mongos with secondaryPreferred after creating a new collection on one mongos Created: 22/Dec/20 Updated: 27/Oct/23 Resolved: 11/Aug/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.0.21 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Chan Lewis | Assignee: | Randolph Tan |
| Resolution: | Works as Designed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Sprint: | Sharding 2021-01-25, Sharding 2021-02-22, Sharding 2021-03-08, Sharding 2021-03-22, Sharding 2021-04-05, Sharding 2021-04-19, Sharding 2021-05-03 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||
| Description |
|
Hi. For some reason we're using Mongo 4.0, and found the following issue. After I created a sharded collection with hashed sharding and inserted some data on mongos A, I cannot read some of data with readPreference=secondary or secondaryPreferred from another mongos B.
I tried mongo shell, python driver, mgo driver, and they all had the same issue. Finally I tried mongo 4.4 on local host, the issue was gone.
I tried to find the issue on Jira, but NOT FOUND. So is there an issue about this ? And I really want to know why that happens.
Thanks. |
| Comments |
| Comment by Randolph Tan [ 28/Jan/21 ] | |||||||||||||||||||
|
I think you forgot to specify the read concern for the read. Can you try it with this:
I also think you are not using causal consistency correctly. To be able to read your own write, you need to use the same session for the both the write and the read. In your script, you are only applying the causal consistency after the write has already occurred. The mongo shell might not have the proper helper for this when you're connecting to different mongos manually like in your example (but you can still do it using low level commands which I prefer not to show you here as they are not public APIs and can subject to change without warning), but most of the official drivers should have support for this. | |||||||||||||||||||
| Comment by Chan Lewis [ 26/Jan/21 ] | |||||||||||||||||||
|
Hi, Randolph I cannot see the result you mentioned with mongo shell.
From mongos B:
Can you make a script example to verify this ? | |||||||||||||||||||
| Comment by Randolph Tan [ 21/Jan/21 ] | |||||||||||||||||||
|
I believe you are seeing this behavior possibly because the read concern was not specified when the read commands were sent. These kinds of commands will behave like read concern available and they will not be subject to shard version checking. I believe that if you set the read concern to something other than available, you should be able to see the expected data, as long as you set your read to be causally consistent. | |||||||||||||||||||
| Comment by Edwin Zhou [ 05/Jan/21 ] | |||||||||||||||||||
|
baiwfg2@gmail.com, thank you for the scripts, they proved to be extremely helpful. I was able to successfully reproduce the issue. I'll be assigning this ticket over to the appropriate team for further evaluation and any updates will be posted on this ticket. Best, | |||||||||||||||||||
| Comment by Chan Lewis [ 05/Jan/21 ] | |||||||||||||||||||
|
Hi. Edwin. Your script is not able to reproduce my case. I modified that into four pieces, you can run like this and would find my results.
| |||||||||||||||||||
| Comment by Edwin Zhou [ 04/Jan/21 ] | |||||||||||||||||||
|
Thanks for the additional details that you provided. I've adjusted my reproduction attempts to reflect them. I remain unable to recreate the issue you're observing after many attempts. I've attached some repro scripts I used that you can run against your mongos. You can run repro.js on mongos A, and read.js on mongos B.
I'm also assigning this to the sharding team so they can review my attempts and see if they have any additional input. Best, | |||||||||||||||||||
| Comment by Chan Lewis [ 30/Dec/20 ] | |||||||||||||||||||
|
Hi. Edwin, I have a big finding ! I'm quite sure it's like a bug, not only on 4.0, but also on 4.4.2 (I downloaded it from binary tarball). Here're I did: After creating two-shard cluster with two mongos, A & B, on mongos A I do the following: create db3.t1 first use db3 db.adminCommand({enablesharding:"db3"}) sh.shardCollection("db3.t1", {uin:"hashed"}) for (var i=1; i<=100; i++) db.t1.insert({uin:i}) Then I check with python program (or mongo shell with secondaryPreferred mode) above with the following script: check all data of db3.t1 on mongos A and mongos B root@setup-mongo$ for i in `seq 1 100`; do a=`python3 check-secondary.py db3 t1 $i`; lines=`echo "$a" | grep 'uin' | wc -l` ; if [ x$lines != x"2" ]; then echo "$i seen from one mongos" ; fi ; done No output, which indicates all data of db3.t1 are found on mongos A and mongos B
Then I create a second sharded collection t2 on db3, sh.shardCollection("db3.t2", {uin:"hashed"}) for (var i=1; i<=100; i++) db.t2.insert({uin:i})
Then Check all data of db3.t2 on mongos A and mongos B root@setup-mongo$ for i in `seq 1 100`; do a=`python3 check-secondary.py db3 t2 $i`; lines=`echo "$a" | grep 'uin' | wc -l` ; if [ x$lines != x"2" ]; then echo "$i seen from one mongos" ; fi ; done ...
As showed in the above picture, 1 cannot be found from mongos B, 3 can.
Keep creating collection t3, and it behaves like case in t2.
All this behavior is strange and not consistent ! We should find all data on mongos A & B no matter what the collection is.
I've uploaded the diagnostic data. I'd like to discuss next.
| |||||||||||||||||||
| Comment by Edwin Zhou [ 29/Dec/20 ] | |||||||||||||||||||
|
It's possible in this circumstance that the routing table cache may have outdated information about the secondary which gets updated after reading from the primary node. We can further investigate by analyzing diagnostics from the mongos and see if an expected refresh was made once the secondary is read on mongos B. Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) and attach it to this ticket? Kind regards, | |||||||||||||||||||
| Comment by Chan Lewis [ 29/Dec/20 ] | |||||||||||||||||||
|
From my perspective in mongo 4.0, it's not about stale data because of async replication. It's about some of data can never be read with readPref=secondary or secondaryPreferred until read once with non-secondary mode. | |||||||||||||||||||
| Comment by Chan Lewis [ 29/Dec/20 ] | |||||||||||||||||||
|
Or you can do like this, write a simple python like the following one, check whether to read all data from B. Finally you'll find there're always some keys that can't be found from mongos B until B is flushed or read with Primary.
| |||||||||||||||||||
| Comment by Chan Lewis [ 29/Dec/20 ] | |||||||||||||||||||
|
Hi Edwin. Thanks for your reply. You mean the problem still exists in mongo 4.4, which I haven't found. It's very easy to reproduce this. Create a sharded cluster, create a sharding collection t: sh.shardcollection("db1.t", k : {k: "hashed"} ) (must be hashed sharding. I tested on range sharding, it didn't happened*)* Then * try to insert a few hundred docs into t: for(var i=1; i<200; i++) db.t.insert({k:i}) *from a mongos A. Then check data with readPreference=secondary or secondaryPreferred (db.getMongo().setReadPref("secondaryPreferred")) from mongos B (before this DONOT do anything with B). After some tries, you'll find some keys can be found from B, some cannot (What I expect is all data can be found like in mongo 4.4) . Finally try to query any data with default readPreference from mongos B in another mongo shell, then switch the previous shell, you'll find all data.
I wonder whether mongo tackled this specific problem explicitly. | |||||||||||||||||||
| Comment by Edwin Zhou [ 28/Dec/20 ] | |||||||||||||||||||
|
Reading with a read preference of anything other than primary has a chance to return stale data, which may explain why you don't see the latest data after insertion when reading from a secondary. In order to further investigate, could you attach the logs during the data insertion and the attempted read to this ticket? Best, Edwin | |||||||||||||||||||
| Comment by Chan Lewis [ 22/Dec/20 ] | |||||||||||||||||||
|
Another important thing: After I read once with Primary from mongos B, then I can read all data from B thereafter with Secondary or SecondaryPreferred. |