[KAFKA-307] Investigate using batch based sizing Created: 06/Apr/22 Updated: 28/Oct/23 Resolved: 10/Jan/23 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | None |
| Fix Version/s: | 1.9.0 |
| Type: | Improvement | Priority: | Unknown |
| Reporter: | Ross Lawley | Assignee: | Ross Lawley |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | size-xsmall | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Quarter: | FY23Q4 |
| Description |
|
Currently, max.batch.size - determines the size of the batch returned from poll. Investigate allowing for the SourceTask.poll method to just return the batch per poll. This would link each poll() call to a getMore() and potentially improve through put of the connector. |
| Comments |
| Comment by Githook User [ 10/Jan/23 ] |
|
Author: {'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}Message: Align change stream cursor and batching with poll batching (#127) Use `poll.await.time.ms` as the `maxAwaitTime` for the cursor
|