[KAFKA-52] MongoDB Kafka Connector > Source Connector Configuration Properties > collection Created: 15/Aug/19  Updated: 07/Oct/19  Resolved: 07/Oct/19

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Cristian Manuel Vertiz Fernandez Assignee: Ross Lawley
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Make "collection" Source Connector Configuration Property as collection white list comma separated



 Comments   
Comment by Ross Lawley [ 07/Oct/19 ]

Hi cristian.vertiz@mojix.com,

Performance wise - matching 50 collections with as single cursor is considered more performant than registering 50 separate Change Stream cursors / kafka connectors. It would use less resources both in the JVM and on the MongoDB side.

Ross

Comment by Cristian Manuel Vertiz Fernandez [ 19/Aug/19 ]

Hi Ross,

Thanks, that worked for me, but still concerned about performance, I mean, if I have a database with 1000 collections, but I want to register events from only 50 collections, how performance is being affected? (otherwise, I would register 50 connectors)

 

Cris.

Comment by Ross Lawley [ 16/Aug/19 ]

Hi cristian.vertiz@mojix.com,

See the event document format - the namespace information is in the ns field. So I think the following should work:

[{"$match": {"ns.coll": {$in : ["coll1", "coll2", ...]}}]

Ross

Comment by Cristian Manuel Vertiz Fernandez [ 16/Aug/19 ]

Hi @Ross Lawley,

Thanks for answering, so you mean filter collections using pipeline property?

Something like (I need to understand correct way): 

 

[{"$match": {"collection": {$in : ["coll1", "coll2", ...]}}]

 

Comment by Ross Lawley [ 16/Aug/19 ]

Hi cristian.vertiz@mojix.com,

Thanks for the ticket. The source connector works off a single change stream cursor, so for the time being to watch multiple collections you would need to watch the whole database and filter the results based on provided namespaces.

Ross

Generated at Thu Feb 08 09:05:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.