[KAFKA-355] poll.await.time.ms in source connector not working ? Created: 23/Feb/23  Updated: 27/Oct/23  Resolved: 23/Feb/23

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: 1.9.1
Fix Version/s: None

Type: Question Priority: Unknown
Reporter: Hoàng Việt Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to KAFKA-356 Update poll.await.time.ms documentation Closed

 Description   

I want to apply Mongo connect  as cronjob, so I change poll.await.time.ms to high number (60000 - 1 minutes or 1 day).

But as soon as I update document, the message produced immediately.

{  "name": "pp",  
"config": {    
"behavior.on.null.values": "ignore",    "connection.timeout.ms": "60000",    
"max.retries": "2147483647",    
"read.timeout.ms": "300000",    "max.connection.idle.time.ms": "3000000",    "behavior.on.malformed.documents": "warn",    "key.converter.schemas.enable": "false",    "value.converter.schemas.enable": "false",   
 "name": "pp",    
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",    "tasks.max": "1",   
 "key.converter": "Json",    
"value.converter": "Json",    
"errors.retry.timeout": "-1",    
"errors.tolerance": "none",    
"connection.uri": "mongodb://...",   
 "database": "...",    
"collection": "...",   
 "pipeline": "[    {       \"$match\":{  \"operationType\":{             \"$in\":[                \"insert\",                \"update\",                \"delete\",                \"replace\"             ]          }       }    } ]",    "publish.full.document.only": "false",    "change.stream.full.document": "updateLookup",    "poll.await.time.ms": "600000",    
"startup.mode": "copy_existing",    
"mongo.errors.tolerance": "none",    
"offset.partition.name": "pp.0"  }} 



 Comments   
Comment by Ross Lawley [ 23/Feb/23 ]

Hi 0124hoang@gmail.com,

Thanks for the ticket, I've added KAFKA-356 to ensure the documentation is updated after the change in KAFKA-306 released in 1.9.0 which simplified the connector and now uses the poll.await.time.ms configuration as the parameter for maxAwaitTimeMS.

This now means poll.await.time.ms is:

The maximum amount of time in milliseconds the server waits for new data changes to report to the change stream cursor before returning an empty batch.

I have opened KAFKA-356 to get the documentation updated.

There may be a Kafka based configuration to limit the time between polls, I'm not sure. Also I'm not aware of any best practices for running Kafka connect via cronjobs as it isn't a use case I'm familiar with. Kafka mailing lists / support may provide further answers.

As the connector works in the way its intended to, I'm closing this ticket.

Ross

Generated at Thu Feb 08 09:06:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.