[KAFKA-115] Connector stop itself after db timeout. Created: 15/Jun/20 Updated: 27/Oct/23 Resolved: 15/Jun/20 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Gil Vander Marcken | Assignee: | Ross Lawley |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Description |
|
We are running a single-instance mongo replica-set, sometimes this replica-set is changing availbility zones. It might take a bit of time for the db to be available again when this occurs (typically 2 minutes).
The only way is then to manually restart the connector :
Is it be possible to configure the behaviour of the source connector when encountering com.mongodb.MongoTimeoutException ? |
| Comments |
| Comment by Ross Lawley [ 16/Jun/20 ] |
|
Hi gilvdm@gmail.com, A longer timeout would amount to being the same as a retry mechanism. The issue here is a MongoTimeoutException could mean an improperly configured connector or a reelection and a primary node change. However, 30 seconds (the default server selection timeout) should be long enough for a reelection of a primary node. I understand that in your case that is isn't but there is configuration you can use for your scenario. Ross |
| Comment by Gil Vander Marcken [ 15/Jun/20 ] |
|
Hi Ross Lawley,
Thank you for your feedback, I did not know about the trick for putting the serverSelectionTimeout in the url of the mongoDB. I don't think I need mongo-db support, the db is not unavailable during a certain amount of time, that's a fact, it's out of their hands, the question is really how will the connector behave in such case. I understand the exception is emanating from com.mongodb inernals but com.mongodb.kafka.connect.source.MongoSourceTask should be able to handle MongoTimeoutException and propose a retry mechanism instead of flat out stopping. For further reference/inspiration the debezium mongodb connector has this feature. |
| Comment by Ross Lawley [ 15/Jun/20 ] |
|
Hi gilvdm@gmail.com, Thanks for your report. For future reference, please note that this project is for reporting bugs or feature suggestions for the driver. For MongoDB-related support discussion please post on the MongoDB Community Forums. A question like this involving more discussion would be best posted on the MongoDB community. You can change the server selection timeout to one that is suitable for your situation via the url eg: mongodb://localhost:27017/?serverSelectionTimeoutMS=60000
Ross |