[SERVER-80713] ID on exhausted cursor no longer 0 Created: 04/Sep/23 Updated: 28/Nov/23 Resolved: 28/Nov/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 7.0.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Cédric Chantepie | Assignee: | Bernard Gorman |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | query-director-triage | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
|||||||||||||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Query Execution
|
|||||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | |||||||||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: | Using mongosh:
With MongoDB 6.0.3; id: Long("0") as exhausted as limit already reached with the first.
With MongoDB 7.0.0; id: Long("2959155384878848610")
|
|||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | ||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Hi, First thanks for the great job with releasing MongoDB 7. Maintaining some tools around MongoDB, we can see that the ID of exhausted cursor is no longer indicated as 0. I cannot see any specific note about it in the changelog/driver compatibility documentation? Is it an intended behaviour (for now that raises as a bug for us) ? |
| Comments |
| Comment by Cédric Chantepie [ 07/Nov/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
That's indeed a significant behaviour change. As for the workaround of omiting the batchSize in case of "conflict" with the limit, as you said, these are set on different layers, so having to correlate them is not trivial, and has impact on the client side. | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Bernard Gorman [ 06/Nov/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Hi chantepiecedric@gmail.com, my apologies for the delay in replying. The reason for the behaviour you're seeing is that the limit and batchSize parameters operate on two completely different layers of the query system; the limit is part of the query execution plan itself, while the batchSize is applied outside of query execution, as we are building the set of results to be returned to the client. When issuing a request whose limit and batchSize are both 1, the limit allows the first document to pass through the query plan, and we return it to the client immediately because the batchSize is 1. But the limit itself is not responsible for closing the cursor; that will only happen when the query plan returns an EOF result to the cursor management code. The cursor is therefore not closed because we haven't yet seen an EOF from the query plan, and we won't until the next time we request a result - that is, on the following getMore. This will happen for any query where limit == batchSize and there are a sufficient number of results to fill the batch. While this is a change in behaviour in SBE on 7.0 when running find queries on a replica set deployment, it's worth noting that this behaviour has always been the case for aggregate operations whose $limit matches their batchSize, and for find queries on mongoS, and there are other similar scenarios elsewhere in the query system where application code should not prematurely assume that the cursor has closed. In general, it is not safe or advisable for client code to stop iterating a cursor until the driver explicitly reports that the cursor has been closed, even if the client app "knows" that it will only ever see a single result, because the query system doesn't know it's going to hit EOF until it actually hits EOF. The fact that the pre-7.0 classic engine's find command exhibits different behaviour on a replica set is an incidental implementation detail, not an intentional part of the API. Note that this question was already considered in In terms of your use-case, the best option here is simply to omit the batchSize from these requests, since batchSize is redundant in any case where it is equal to or greater than the limit. Omitting the batchSize will cause the batch builder to immediately request a second result from the query plan, which will in turn cause the limit stage to return EOF, allowing the server to return the single result and close the cursor in the same operation. If for some reason omitting batchSize is not an option, you can use the runCommand API to issue a getMore on the cursor to exhaust it, or use our drivers' cursor API to iterate the cursor until it closes, or issue a killCursors command to simply erase the cursor from the server. Hope this helps to clarify things! Bernard | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Cédric Chantepie [ 03/Nov/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
About two month later, for something quite blocking. Is there any news? | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Cédric Chantepie [ 19/Oct/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Any news ? | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 09/Sep/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Thank you chantepiecedric@gmail.com for reaching out to us. The non-zero cursor ID indicates the cursor has not been exhausted and that there is still in-memory state associated with the cursor on the MongoDB Server side in its CursorManager. A non-zero cursor ID does not always imply there are more results to fetch. The behavior you are seeing where the cursor is not exhausted is related to how the new Slot-Based Query Execution Engine handles the combination of the limit and batchSize parameters. When the batchSize parameter is omitted, the cursor will be exhausted as part of the initial find command. However, when the batchSize parameter is present together with an equal limit, the cursor will be exhausted only as part of a subsequent getMore command. This ticket has been assigned to the Query team who are investigating whether there is a suitable solution to this behavior difference. Please continue watching this ticket for an update.
| |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Cédric Chantepie [ 09/Sep/23 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Is there any news? That's quite a blocker. |