[SERVER-27170] A possibility to read data from currently available shards, ignoring unavailable. Created: 23/Nov/16  Updated: 06/Dec/22  Resolved: 23/Nov/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Viacheslav Kulyk Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-19842 Support allowPartialResults in new mo... Closed
duplicates SERVER-2020 option to mongos to allow partial res... Closed
Assigned Teams:
Sharding
Participants:

 Description   

At the moment, having e.g. 10 shards, each tagged to some geo location (tag aware sharding), if I have no connection to replicas of at least 1 shard (global query w/o tag), I cannot read any data at all (get error that one shard replicas are not available).
For our business flow, it would be fine to see at least available shards data, ignoring that one unavailable.
Could you please add some parameter to find operation (e.g. ignoreUnavailableShards=true), allowing to do such kind of reads?



 Comments   
Comment by John Gibbons [ 11/Oct/17 ]

Also, I don't think it works fully as expected. Raised jira SERVER-31511

Comment by John Gibbons [ 11/Oct/17 ]

AllowPartialResults still not documented as at version 3.6 of mongo docs. Have raised JIRA.
https://jira.mongodb.org/browse/DOCS-10886

Comment by Andy Schwerin [ 24/Nov/16 ]

If there is no sort specified, then mongos starts returning data from each shard as soon as it responds. a low batch size will keep mongos for waiting on data from a second shard when it has received data from a first shard, because the first shard will typically produce a full batch.

Comment by Viacheslav Kulyk [ 24/Nov/16 ]

The case is that one/few shards are not reachable, so batch size will not help. What would help is a possibility to set a waitTime for the query and to return what is got after the waitTime expired. Is that possible?

Comment by Andy Schwerin [ 23/Nov/16 ]

I believe that if your request has no sort order, beginning in 3.2, results should come back as soon as they're available, provided you've got at least one full batch to deliver. If you set your batch size for the first batch to a small-ish number of documents, such that any shard is likely to return that many results, you might just be fine. It's not a perfect solution, but it might work.

Comment by Kaloian Manassiev [ 23/Nov/16 ]

The code for the partial results is here. Essentially, it treats errors coming from fetching results from shards as if the shard has no data.

Unfortunately, there is no way to deterministically control how long it would be until the query gives up on a given shard. For fast network errors, such as HostUnreachable it might be discovered quickly, however if connection establishment takes longer it may be up to 5 seconds. In addition, query has a retry logic, which would retry the query up to 3 times before it gives up.

Comment by Viacheslav Kulyk [ 23/Nov/16 ]

Thank you.
The reason of lacking a connectivity is: some small office does not want to have replicas of all other offices (weak hardware), but still wants, in case if it becomes disconnected, to read at least its own shard data or data from shards he does have replica locally. E.g., being disconnected, I still want to read my office users and users from central office, and do not care about users from other secondary offices.

In turn, could you please provide more details about the algorithm the partial works: mongos tries to query all shards - waits for some time (some timeout parameter ?) - returns the data it has after timeout. We need this query to work not longer that 100ms / configurable.

Comment by Kaloian Manassiev [ 23/Nov/16 ]

Thanks, viacheslav.kulyk@avid.com.

It looks like the find operation in the Java Driver has a partial property which can be set to allow for partial results to be retrieved.

Just as a warning, using this property is not recommended because it will silently ignore failed/unreachable shards and your application will have no way of knowing what information is being omitted.

For our information, can you please tell us a bit about your use case and what are the reasons you are lacking connectivity to all nodes?

Comment by Viacheslav Kulyk [ 23/Nov/16 ]

Oh, great. Using:

       <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongo-java-driver</artifactId>
            <version>3.2.2</version>
        </dependency>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongodb-driver-async</artifactId>
            <version>3.2.2</version>
        </dependency>

Comment by Kaloian Manassiev [ 23/Nov/16 ]

Hi viacheslav.kulyk@avid.com,

The way to force sharding to ignore failed/unavailable shards is to use the allowPartialResults:true query/find command option. Unfortunately I see that there is no documentation for it and I will file a ticket to get it documented.

If you are running queries through the shell, it would look something like this:

db.coll.find({ <query> }).allowPartialResults();

Outside of the shell, the way this option is exposed varies across different drivers. Which MongoDB driver are you using?

Best regards,
-Kal.

Generated at Thu Feb 08 04:14:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.