[SERVER-53474] Cannot read from another mongos with secondaryPreferred after creating a new collection on one mongos Created: 22/Dec/20  Updated: 27/Oct/23  Resolved: 11/Aug/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.21
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Chan Lewis Assignee: Randolph Tan
Resolution: Works as Designed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File 1-create-t1-from-A.js     File 2-read-t1-from-B.js     File 3-create-t2-from-A.js     File 4-read-t2-from-B.js     PNG File Screen Shot 2021-01-06 at 1.26.47 PM.png     File diagnostic.data-on-4.4.2.tar.bz2     PNG File image-2020-12-30-21-22-38-181.png     PNG File image-2021-01-05-11-08-14-899.png     PNG File image-2021-01-26-10-59-01-238.png     PNG File image-2021-01-26-10-59-27-147.png     PNG File image-2021-01-26-11-18-04-528.png     PNG File mdb 4.0.21.png     PNG File mdb 4.4.2.png     PNG File read col1.png     File read.js     File repro.js     File repro.tar.gz     PNG File skip reading col1.png     PNG File user mdb 4.4.2.png    
Issue Links:
Duplicate
is duplicated by SERVER-54373 read from mongos with readPref=second... Closed
is duplicated by SERVER-54625 Maybe the root case for reading from ... Closed
is duplicated by SERVER-76541 documents can't be find with secondar... Closed
Sprint: Sharding 2021-01-25, Sharding 2021-02-22, Sharding 2021-03-08, Sharding 2021-03-22, Sharding 2021-04-05, Sharding 2021-04-19, Sharding 2021-05-03
Participants:
Case:

 Description   

Hi. For some reason we're using Mongo 4.0, and found the following issue.

After I created a sharded collection with hashed sharding and inserted some data on mongos A,  I cannot read some of data with readPreference=secondary or secondaryPreferred from another mongos B.

 

I tried mongo shell, python driver, mgo driver, and they all had the same issue.

Finally I tried mongo 4.4 on local host, the issue was gone.

 

I tried to find the  issue on Jira, but NOT FOUND. So is there an issue about this ? And I really want to know why that happens.

 

Thanks.



 Comments   
Comment by Randolph Tan [ 28/Jan/21 ]

I think you forgot to specify the read concern for the read. Can you try it with this:

db.t5.find().readConcern("local").itcount();

I also think you are not using causal consistency correctly. To be able to read your own write, you need to use the same session for the both the write and the read. In your script, you are only applying the causal consistency after the write has already occurred. The mongo shell might not have the proper helper for this when you're connecting to different mongos manually like in your example (but you can still do it using low level commands which I prefer not to show you here as they are not public APIs and can subject to change without warning), but most of the official drivers should have support for this.

Comment by Chan Lewis [ 26/Jan/21 ]

Hi, Randolph

I cannot see the result you mentioned with mongo shell.

 

From mongos B:

 

Can you make a script example to verify this ?

Comment by Randolph Tan [ 21/Jan/21 ]

I believe you are seeing this behavior possibly because the read concern was not specified when the read commands were sent. These kinds of commands will behave like read concern available and they will not be subject to shard version checking. I believe that if you set the read concern to something other than available, you should be able to see the expected data, as long as you set your read to be causally consistent.

Comment by Edwin Zhou [ 05/Jan/21 ]

baiwfg2@gmail.com, thank you for the scripts, they proved to be extremely helpful. I was able to successfully reproduce the issue. I'll be assigning this ticket over to the appropriate team for further evaluation and any updates will be posted on this ticket.

Best,
Edwin

Comment by Chan Lewis [ 05/Jan/21 ]

Hi. Edwin. Your script is not able to reproduce my case. I modified that into four pieces, you can run like this and would find my results.

 

Comment by Edwin Zhou [ 04/Jan/21 ]

Hi baiwfg2@gmail.com,

Thanks for the additional details that you provided. I've adjusted my reproduction attempts to reflect them.

I remain unable to recreate the issue you're observing after many attempts. I've attached some repro scripts I used that you can run against your mongos.

You can run repro.js on mongos A, and read.js on mongos B.

mongo --port 27017 repro.js
mongo --port 27018 read.js

I'm also assigning this to the sharding team so they can review my attempts and see if they have any additional input.

Best,
Edwin

Comment by Chan Lewis [ 30/Dec/20 ]

Hi. Edwin,

I have a big finding ! I'm quite sure it's like a bug, not only on 4.0, but also on 4.4.2 (I downloaded it from binary tarball). Here're I did:

After creating two-shard cluster with two mongos, A & B, on mongos A I do the following:

create db3.t1 first

use db3

db.adminCommand({enablesharding:"db3"})

sh.shardCollection("db3.t1", {uin:"hashed"})

for (var i=1; i<=100; i++) db.t1.insert({uin:i})

Then I check with python program (or mongo shell with secondaryPreferred mode) above with the following script:

check all data of db3.t1 on mongos A and mongos B

root@setup-mongo$ for i in `seq 1 100`; do a=`python3 check-secondary.py db3 t1 $i`; lines=`echo "$a" | grep 'uin' | wc -l` ; if [ x$lines != x"2" ]; then echo "$i seen from one mongos" ; fi ; done

No output, which indicates all data of db3.t1 are found on mongos A and mongos B

 

Then I create a second sharded collection t2 on db3,

sh.shardCollection("db3.t2", {uin:"hashed"})

for (var i=1; i<=100; i++) db.t2.insert({uin:i})

 

Then

Check all data of db3.t2 on mongos A and mongos B

root@setup-mongo$ for i in `seq 1 100`; do a=`python3 check-secondary.py db3 t2 $i`; lines=`echo "$a" | grep 'uin' | wc -l` ; if [ x$lines != x"2" ]; then echo "$i seen from one mongos" ; fi ; done
1 seen from one mongos
2 seen from one mongos
4 seen from one mongos
5 seen from one mongos
7 seen from one mongos
9 seen from one mongos
10 seen from one mongos
13 seen from one mongos
14 seen from one mongos
15 seen from one mongos
16 seen from one mongos

...

As showed in the above picture, 1 cannot be found from mongos B, 3 can.

 

Keep creating collection t3, and it behaves like case in t2.

 

All this behavior is strange and not consistent ! We should find all data on mongos A & B no matter what the collection is. 

 

I've uploaded the diagnostic data. I'd like to discuss next.

 

Comment by Edwin Zhou [ 29/Dec/20 ]

Hi baiwfg2@gmail.com,

It's possible in this circumstance that the routing table cache may have outdated information about the secondary which gets updated after reading from the primary node. We can further investigate by analyzing diagnostics from the mongos and see if an expected refresh was made once the secondary is read on mongos B. Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) and attach it to this ticket?

Kind regards,
Edwin

Comment by Chan Lewis [ 29/Dec/20 ]

From my perspective in mongo 4.0, it's not about stale data because of async replication. It's about some of data can never be read with readPref=secondary or secondaryPreferred until read once with non-secondary mode.

Comment by Chan Lewis [ 29/Dec/20 ]

Or you can do like this, write a simple python like the following one, check whether to read all data from B. Finally you'll find there're always some keys that can't be found from mongos B until B is flushed or read with Primary.

 

// code placeholder
def test(dbname, colname, ii):
    # mongos A
    uri = "mongodb://admin:admin@9.134.5.74:30001/admin?readPreference=secondary" 
    print("read from ", uri)                                                                                                                
    x = MongoClient(uri)                                                                                                                 
    db = x[dbname]
    co = db[colname]                                                                                                                    
    for item in co.find({"uin":ii}):                                                                                  
        print(item, "\n")   
                        
    # mongos B                                                                            
    uri = "mongodb://admin:admin@9.134.5.74:30002/admin?readPreference=secondary"       
    print("read from ", uri)                                                                                                    
    x = MongoClient(uri)                                                                                                                        
    db = x[dbname]   
    co = db[colname]                                                                                                                     
    for item in co.find({"uin":ii}):
        print(item, "\n")

Comment by Chan Lewis [ 29/Dec/20 ]

Hi Edwin. Thanks for your reply. You mean the problem still exists in mongo 4.4, which I haven't found. 

It's very easy to reproduce this. Create a sharded cluster, create a sharding collection t: sh.shardcollection("db1.t", k : {k: "hashed"} ) (must be hashed sharding. I tested on range sharding, it didn't happened*)*

Then * try to insert a few hundred docs into t: for(var i=1; i<200; i++) db.t.insert({k:i}) *from a mongos A. Then check data with readPreference=secondary or secondaryPreferred (db.getMongo().setReadPref("secondaryPreferred")) from mongos B (before this DONOT do anything with B). After some tries, you'll find some keys can be found from B, some cannot (What I expect is all data can be found like in mongo 4.4) . Finally try to query any data with default readPreference from mongos B in another mongo shell, then switch the previous shell, you'll find all data.

 

I wonder whether mongo tackled this specific problem explicitly.

Comment by Edwin Zhou [ 28/Dec/20 ]

Hi baiwfg2@gmail.com,

Reading with a read preference of anything other than primary has a chance to return stale data, which may explain why you don't see the latest data after insertion when reading from a secondary. 

In order to further investigate, could you attach the logs during the data insertion and the attempted read to this ticket? 

Best,

Edwin

Comment by Chan Lewis [ 22/Dec/20 ]

Another important thing: After I read once with Primary from mongos B, then I can read all data from B thereafter with Secondary or SecondaryPreferred.

Generated at Thu Feb 08 05:31:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.