[SERVER-38081] Query with `secondary` read preference don't use SHARDING_FILTER Created: 12/Nov/18  Updated: 27/Oct/23  Resolved: 14/Nov/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.8, 4.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Artem Assignee: Danny Hatcher (Inactive)
Resolution: Works as Designed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Steps for reproduce:

  1. Start cluster with 2 replicasets with 3 nodes in each;
  2. Run commands:

    > use test
    > db.foo.insert({_id: 1, a: 1})
    > sh.enableSharding("test")
    > sh.shardCollection("test.foo", { a: 1 })
    > sh.splitAt( "test.foo", { a: 10 } ) 

  1. Wait few seconds for balancing `test.foo`

> sh.status()
                        { "a" : { "$minKey" : 1 } } -->> { "a" : 10 } on : rs1 Timestamp(2, 0) 
                        { "a" : 10 } -->> { "a" : { "$maxKey" : 1 } } on : rs0 Timestamp(2, 1)  

  1. Run commands:

    > db.foo.find()
    { "_id" : 1, "a" : 1 }
    > db.getMongo().setReadPref('secondary')
    > db.foo.find()
    { "_id" : 1, "a" : 1 }
    { "_id" : 1, "a" : 1 }

     

Expected result: second db.foo.find() returns exactly one record.

Actual result: second db.foo.find() return two records (one orphaned records from `rs0` shard and actual record from `rs1` shard).

Participants:

 Description   

Query on `secondary` don't use SHARDING_FILTER stage, although `secondary` have all data for SHARDING_FILTER in `cache` database.

I understand, that I can get inconsistent data from `secondary`, but I expected inconsistency by in-progress chunk moving and oplog delay.

In this case I got permanent inconsistency event on up-to-date replicasets with stopped balancer. Also I see very old orphaned documents (above week age).



 Comments   
Comment by Danny Hatcher (Inactive) [ 14/Nov/18 ]

As this behavior is expected, I will close this ticket. If you have a recommendation to improve our documentation, please feel free to create a DOCS ticket with the improvements you would like to see made.

Comment by Sergey Zagursky [ 13/Nov/18 ]

It looks like that the documentation should be updated here with explicit mention that sharding filter is not applied with `available` read concern.

Comment by Andy Schwerin [ 12/Nov/18 ]

The behavior of never consulting the chunk ownership and "available" read concern is intentional, to ensure that data can be returned even if the node cannot contact other nodes in the cluster.

Comment by Artem [ 12/Nov/18 ]

This issue is reproducing only with `available` read concern (no matter: SECONDARY or PRIMARY).

However, a query with "available" read concern may return orphan documents if the shard is undergoing chunk migrations since the "available" read concern.

The problem is that with `available` read concern always return orphan documents, because shard never check chunk ownership (but shard have chunk metadata in `config` database).

Comment by Danny Hatcher (Inactive) [ 12/Nov/18 ]

Hello Artem,

As you mentioned in your followup comment, this is a function of how a read concern of "available" works in sharded clusters.

In MongoDB 3.6, read concern "available" is used by default for any secondary reads. In sharded clusters, "available" can return orphaned documents if chunk migrations are occurring. In order to prevent this from happening, we recommend specifying a read concern with your queries or associate them with causally consistent sessions. Either option will ensure that read concern "available" is not used and you should not experience this situation again.

If you specify a different read concern or test causally consistent sessions, do you still see the issues?

Thank you,

Danny

Comment by Artem [ 12/Nov/18 ]

I found, that this issue can be reproduced on `primary` with read concern `available`.

Generated at Thu Feb 08 04:47:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.