[KAFKA-392] Support for multiple clusters on one source connector Created: 05/Dec/23 Updated: 01/Feb/24 Resolved: 01/Feb/24 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Unknown |
| Reporter: | Florin Bunău | Assignee: | Robert Walters |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
Hello! Thanks for the multiplexing capabilities by being able to specify multiple collections and also multiple topics to fan that out into. Have a question/request related to multiplexing multiple collections located in multiple clusters. I have not seen that as a possibility as only 1 URI is allowed to be configured.
Is there a special way I am missing to configure the URI such that multiple clusters are accessible to the changestream? is there some Mongo technology that can be stood up as an aggregator ?
Use case: For replication purposes you are forced to deploy 1 source connector per mongo cluster. but that is overkill as there is not a lot of volume and we have a lot of clusters. Expected: Ability to connect to multiple mongo clusters as a source |
| Comments |
| Comment by Florin Bunău [ 31/Jan/24 ] |
|
Hi Robert, I appreciate the details. yep i imagined this might be the case. thank you for the confirmation |
| Comment by Robert Walters [ 31/Jan/24 ] |
|
Today, change streams only work within a single cluster, which is why the kafka connector can only support a single cluster. To add multiple listeners to the Kafka connector for a single instance isn't something that would come easy as it would require storing multiple resume tokens and dealing with error conditions where one cluster might be offline but the others are online, or similar scenarios where one resume token from a cluster might have fallen off the oplog but the others are fine so do you fail the connector as a whole or just that cluster, etc.. It would be a very large work item and given the amount of other tickets that are requested, I do not see this as something we can implement in the connector. The scenario you are describing might be better handled with an ETL tool that is designed to handle this multiplexing. |
| Comment by Florin Bunău [ 31/Jan/24 ] |
|
Hello. any updates on this ? Do you think it would be possible ? |
| Comment by PM Bot [ 05/Dec/23 ] |
|
Hi florin@leadiq.com, thank you for reporting this issue! The team will look into it and get back to you soon. |