[DRIVERS-1589] Allow changing BatchSize for getMores; needed when BatchSize(0) uses default size Created: 01/Mar/21 Updated: 24/Feb/23 |
|
| Status: | Backlog |
| Project: | Drivers |
| Component/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Lorenz Huelsbergen | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Description |
| Comments |
| Comment by Lorenz Huelsbergen [ 15/Mar/21 ] |
|
(fyi- i was not receiving updates to the ticket for a bit it seems, but it looks like proper permissions have now been restored.) as far as more context: ADL needs to two things ideally when running an aggregation from a driver: 1) immediate access to the cursor w/o any docs; and 2) the ability to set a batch size other than the default. point 1 is necessary given ADL's model where an ADL client query may be interrupted/canceled at any point by the client. we do this on the Atlas side by killing a cursor. therefore, we need the cursor immediately and cannot wait for the first batch of docs to return. point 2 is necessary because sometimes the batch size is too large and ADL receives too many docs which cannot be processed (consider: many small writes to S3) before the Atlas cursor expires (10m i believe). if we can set a batch size we can circumvent this. it would be ideal to be able to set the batch size dynamically as the query progresses; however, setting it once statically should be sufficient for us to avoid this scenario. currently, points 1 & 2 are mutually exclusive since point 1 is conveyed with a batch size of zero, but this disallows setting a batch size smaller than the default. conversely, setting a batch size smaller than the default (but not zero) precludes getting a cursor immediately. there are many ways to solve this on the driver side. for example, provide a separate aggregation option to get the cursor w/o any docs and keep BatchSize for setting batch sizes other than the default. glad to provide more info where needed; please let me know.
|
| Comment by Alexander Golin (Inactive) [ 15/Mar/21 ] |
|
lorenz.huelsbergen, driver leads are discussing in triage and are wondering if you could update this ticket with more detail about the motivation for this request. Thank you! |
| Comment by Jeffrey Yemin [ 09/Mar/21 ] |
|
The request is specific to ADL's use of the Go driver to communicate with backing Atlas clusters. ADL does not need this capability in all drivers (Note that the issue was originally opened as Furthermore, there is the possibility to expose this feature through the existing unstable BatchCursor API in the Go driver. If this is time-sensitive for ADL, I suggest that we explore that route. |