[KAFKA-307] Investigate using batch based sizing Created: 06/Apr/22  Updated: 28/Oct/23  Resolved: 10/Jan/23

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.9.0

Type: Improvement Priority: Unknown
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: size-xsmall
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Quarter: FY23Q4

 Description   

Currently, max.batch.size - determines the size of the batch returned from poll.

Investigate allowing for the SourceTask.poll method to just return the batch per poll. This would link each poll() call to a getMore() and potentially improve through put of the connector.



 Comments   
Comment by Githook User [ 10/Jan/23 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Align change stream cursor and batching with poll batching (#127)

Use `poll.await.time.ms` as the `maxAwaitTime` for the cursor
Use getMore batches for polling up to `poll.max.batch.size`.

KAFKA-306
KAFKA-307
Branch: master
https://github.com/mongodb/mongo-kafka/commit/013d09fe275a4402680ffde7e32480d2482184b4

Generated at Thu Feb 08 09:06:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.