[SERVER-57863] CursorNotFound since we doubled the number of shards Created: 21/Jun/21 Updated: 03/Nov/21 Resolved: 04/Aug/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Distributed Query Planning, Sharding |
| Affects Version/s: | 4.0.19 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Kay Agahd | Assignee: | Mihai Andrei |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | cursor, query, sharding, timeout | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Query Execution 2021-07-12, Query Execution 2021-07-26, QE 2021-08-09 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Since we added 4 new shards to an existing 5 shard cluster, one of our database clients receives CursorNotFound errors from time to time. The query in mongo-shell syntax looks like this:
The index is defined as follows:
The client application calls constantly getMore (always within the cursorTimeoutMillis of 2 hours). As you can see in the strace of the client, which is running only one query at the time, at 11:29:04 there was a successful getMore call, and one second later the next getMore call failed with CursorNotFound:
The size of one returned document is 100 Bytes on average. The index size is round about 1 GB, all shards together. We found out, that cursors timed out on all nine mongodb shards within a few seconds:
Is it possible that the cursor which the client uses is completely disassociated from the cursors between router and the shards? And if there's some caching or queueing between router and shard nodes, then it could be possible that the "backend" cursors less often get the getMore commands, causing the timeouts. Hypothesis: Each shard returns its batch to the router. The router caches these batches. Batches are fix in size. Since our cluster has nearly doubled the number of shards, the router has now nearly the doubled number of documents in its cache. Thus, the router takes almost twice as long to request the next batch from the shards, which may exceed the cursor timeout. |
| Comments |
| Comment by Kay Agahd [ 03/Nov/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi mihai.andrei, sorry for my late reply, I oversaw your question about the shardkey. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mihai Andrei [ 04/Aug/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi kay.agahd@idealo.de and eric.sedor; I took a look at this ticket and found that it can be fixed with | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mihai Andrei [ 22/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I think the simplest way to fix this would be to tie the query to a session so that the shard cursors don’t time out. After taking a look, I think this could certainly be fixed by As far as the relation to
Do we know anything about the shard key being used/distribution of data in the above? If the data is sharded on clickCount (or even the key pattern of the index), then it's likely related. Regardless of the relation to | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 29/Jun/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
You're right that the batch size workaround won't scale well with shard count; currently we do intend to backport For working with batch size to get the best results in your specific environment I'd encourage you to explore this with our community by posting on the MongoDB Developer Community Forums. I believe it is the case that | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kay Agahd [ 29/Jun/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi eric.sedor, thanks for confirming our hypothesis. By the way, after having added our 4 new shards to the cluster, we also hit the bug solved in The tickets you linked are indeed related, especially The author of
mihai.andrei found out in
mihai.andrei concludes that's rather a feature request than a bug and closes the ticket. Now, that you have a real life use case, even without sort, you may have a reason to solve this issue and backport it to at least version 4 because v5 has not even been published yet.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Eric Sedor [ 28/Jun/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Another way to work around your issue might be to test explicit and smaller batch size options that will prompt more getMores to the shards. The mongos does maintain its own cursors to shards, and there is an inherent buffer in that batch size is not varied by the mongos when it makes requests to shards. So, batches from each cursor to shard members won't be fully consumed by a single getmore to the driver-facing cursor. Doubling the number of shards would certainly have an effect on this. I believe what you're reporting is similar in shape to both Sincerely, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kay Agahd [ 21/Jun/21 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
To work around this issue, the database client now retrieves all documents from the database and keeps them in memory. The query took seven minutes and returned 4618504 documents. After that, these documents will be processed which will take approximately 20 hours. |