[SERVER-54863] Support noCursorTimeout for listCollections command Created: 01/Mar/21 Updated: 26/Mar/21 Resolved: 26/Mar/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Lawrence Tan (Inactive) | Assignee: | James Wahlin |
| Resolution: | Won't Do | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Sprint: | Query Optimization 2021-03-22, Query Optimization 2021-04-05 |
| Participants: |
| Description |
|
https://docs.mongodb.com/manual/reference/command/listCollections/ doesn't have a way to specify noCursorTimeout. In CLOUDP-51006, our use case is to loop through all the collections and process them as we go. However processing each collection might take a long time so we might get hit with a cursor idle timeout. As a workaround today, we are storing all the collections in memory. But that's inefficient for customers with a huge amount of collections. |
| Comments |
| Comment by James Wahlin [ 26/Mar/21 ] |
|
Closing ticket as we do not plan to add noCursorTimeout for the listCollections command. |
| Comment by Lawrence Tan (Inactive) [ 25/Mar/21 ] |
|
Thanks divjot.arora james.wahlin! We do need to support < 5.0. Doesn't sound like we have a simple solution. I think we can close this ticket for now and revisit later if needed. |
| Comment by James Wahlin [ 22/Mar/21 ] |
|
Another potential not-yet-implemented solution would be to implement SERVER-16521, which requests a $listCollections aggregation source/stage. This would allow for alternate pagination schemes that don't rely on maintaining an open cursor. lawrence.tan let me know if you would like a call to discuss. I think there is a reasonable solution for 5.0 but how to handle server-side for previous versions is not clear. |
| Comment by Lawrence Tan (Inactive) [ 11/Mar/21 ] |
|
It seems like refreshSessions require passing in session UUIDs? It doesn't seem like listCollections return any session UUID: https://docs.mongodb.com/manual/reference/command/listCollections/. |
| Comment by James Wahlin [ 10/Mar/21 ] |
|
lawrence.tan / craig.schwarz / jacob.lu - I have a proposal for an alternate way of handling this issue. As you mention, traditionally users have relied on "noCursorTime" to address the case where there may be 10 or more minutes between cursor requests. This solution suffers from the risk that a cursor may be orphaned and will reside in memory until server restart. For MongoDB 5.0 we removed the 10 minute cursor timeout mechanism for any operation that is run under a session. This work was done under |
| Comment by Jacob Lu (Inactive) [ 02/Mar/21 ] |
|
Perhaps one workaround is to batch up the query with the filter on "info.uuid" field. |
| Comment by Craig Schwarz (Inactive) [ 02/Mar/21 ] |
|
the other thing I was unsure of working with this command - even though it returns a cursor it seemed like it was only ever returning all the collections in one batch |