[KAFKA-52] MongoDB Kafka Connector > Source Connector Configuration Properties > collection Created: 15/Aug/19 Updated: 07/Oct/19 Resolved: 07/Oct/19 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Cristian Manuel Vertiz Fernandez | Assignee: | Ross Lawley |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
Make "collection" Source Connector Configuration Property as collection white list comma separated |
| Comments |
| Comment by Ross Lawley [ 07/Oct/19 ] | |
|
Performance wise - matching 50 collections with as single cursor is considered more performant than registering 50 separate Change Stream cursors / kafka connectors. It would use less resources both in the JVM and on the MongoDB side. Ross | |
| Comment by Cristian Manuel Vertiz Fernandez [ 19/Aug/19 ] | |
|
Hi Ross, Thanks, that worked for me, but still concerned about performance, I mean, if I have a database with 1000 collections, but I want to register events from only 50 collections, how performance is being affected? (otherwise, I would register 50 connectors)
Cris. | |
| Comment by Ross Lawley [ 16/Aug/19 ] | |
|
See the event document format - the namespace information is in the ns field. So I think the following should work:
Ross | |
| Comment by Cristian Manuel Vertiz Fernandez [ 16/Aug/19 ] | |
|
Hi @Ross Lawley, Thanks for answering, so you mean filter collections using pipeline property? Something like (I need to understand correct way):
| |
| Comment by Ross Lawley [ 16/Aug/19 ] | |
|
Thanks for the ticket. The source connector works off a single change stream cursor, so for the time being to watch multiple collections you would need to watch the whole database and filter the results based on provided namespaces. Ross |