[KAFKA-288] Using brew installs on mac doesn't seem to allow mongo connector installation Created: 02/Feb/22  Updated: 27/Oct/23  Resolved: 10/Feb/22

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Unknown
Reporter: Alexander Boese Assignee: Robert Walters
Resolution: Gone away Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

 

$ confluent-hub install mongodb/kafka-connect-mongodb:1.6.1
Unable to detect Confluent Platform installation. Specify --component-dir and --worker-configs explicitly. 
Error: Invalid options or arguments 

I realize there's about 50 things I could do wrong to get here. Adding a --component-dir parameter gets further but still doesn't run, and a worker-configs option I don't believe does anything particularly useful, so I think that's a general error that hasn't been paired off to be very specific in this instance.

 

I realize this is not for production, but was just trying a basic setup on Mac for testing. I have downloaded the component, but I'm missing a good walk through. There are some directories that need to be set up in profile according to some postings, but that has not seemed to help either. Is there a good page I can go to that would be a suitable walk through for this? And to be very clear, the connector itself needs to be downloaded unzipped and placed in the right place. The primary necessary components as far as I can tell have been downloaded and installed with brew. 

 



 Comments   
Comment by Alexander Boese [ 10/Feb/22 ]

It looks like some of the connector name have maybe changed, and that is leading to the mix up. That's one thing....but I'll figure out the others.
Just close please.

Thank you.

Mr. Alex Boese
CACI Inc. Federal
O: (202) 436-9888 (skype-based)
H: (443) 825-5760
alexander.boese@caci.com
alexander.a.boese.ctr@mail.mil
________________________________
From: Robert Walters (Jira) <jira@mongodb.org>
Sent: Thursday, February 10, 2022 12:30 PM
To: Boese, Alexander - US <Alexander.Boese@caci.com>
Subject: [MongoDB-JIRA] (KAFKA-288) Using brew installs on mac doesn't seem to allow mongo connector installation

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.

https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png
[cid:jira-generated-image-avatar-aec42e1c-997e-4029-a2ab-c5e4fe88a503] Robert Walters<https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.walters> commented on [Task] KAFKA-288<https://jira.mongodb.org/browse/KAFKA-288>

Re: Using brew installs on mac doesn't seem to allow mongo connector installation<https://jira.mongodb.org/browse/KAFKA-288>

then just copy the JAR file of whatever connector you want to install into the plug in path of your connector. It most likely is
usr/share/java or /usr/share/confluent-hub-components
restart kafka connect and it should show up, you can see the list via
curl --silent http://connect:8083/connector-plugins

[Add Comment] <https://jira.mongodb.org/browse/KAFKA-288#add-comment> Add Comment<https://jira.mongodb.org/browse/KAFKA-288#add-comment>

This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to jira.mongodb.org<https://jira.mongodb.org> using your JIRA, MongoDB Cloud Manager, or MongoDB Atlas credentials.

________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

Comment by Robert Walters [ 10/Feb/22 ]

then just copy the JAR file of whatever connector you want to install into the plug in path of your connector.  It most likely is 
usr/share/java or /usr/share/confluent-hub-components
restart kafka connect and it should show up, you can see the list via 
curl --silent http://connect:8083/connector-plugins 

Comment by Alexander Boese [ 10/Feb/22 ]

I mean most of what I've gotten working has actually been manual on a stand alone instance. Just wanting to continue with that for mongo connector as possible.

Mr. Alex Boese
CACI Inc. Federal
O: (202) 436-9888 (skype-based)
H: (443) 825-5760
alexander.boese@caci.com
alexander.a.boese.ctr@mail.mil
________________________________
From: Robert Walters (Jira) <jira@mongodb.org>
Sent: Thursday, February 10, 2022 12:10 PM
To: Boese, Alexander - US <Alexander.Boese@caci.com>
Subject: [MongoDB-JIRA] (KAFKA-288) Using brew installs on mac doesn't seem to allow mongo connector installation

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.

https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png
[cid:jira-generated-image-avatar-5e771653-c7c6-4b2e-ba9a-e7b483f394fc] Robert Walters<https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.walters> commented on [Task] KAFKA-288<https://jira.mongodb.org/browse/KAFKA-288>

Re: Using brew installs on mac doesn't seem to allow mongo connector installation<https://jira.mongodb.org/browse/KAFKA-288>

I think there is some confusion with regard to what it means to install a connector on Kafka Connect.

What I am suggesting is use docker to launch Kafka Connect and the rest of the kafka services. This just helps with cleanup as nothing is left installed on your mac.

within Kafka Connect you can install as many connectors as you'd like MongoDB, MySQL and whatever other connectors you'd like to use. It doesn't have anything to do with Docker at that point they are just JARs added to a folder on that Kafka Connect instance.

if it is easier I'd be happy to jump on a zoom and go over this, just lmk and Ill email you directly.

[Add Comment] <https://jira.mongodb.org/browse/KAFKA-288#add-comment> Add Comment<https://jira.mongodb.org/browse/KAFKA-288#add-comment>

This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to jira.mongodb.org<https://jira.mongodb.org> using your JIRA, MongoDB Cloud Manager, or MongoDB Atlas credentials.

________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

Comment by Alexander Boese [ 10/Feb/22 ]

Asking for those other things might be a stretch...okay I get that. Is there at least a known working properties file for mongo connector that you can point to? I see a slew of different examples and none of them as far as I can tell are working for me. Maybe tell me which is good for whatever is recent, and I can pick at the problems I am having locally loading that?

Mr. Alex Boese
CACI Inc. Federal
O: (202) 436-9888 (skype-based)
H: (443) 825-5760
alexander.boese@caci.com
alexander.a.boese.ctr@mail.mil
________________________________
From: Robert Walters (Jira) <jira@mongodb.org>
Sent: Thursday, February 10, 2022 8:25 AM
To: Boese, Alexander - US <Alexander.Boese@caci.com>
Subject: [MongoDB-JIRA] (KAFKA-288) Using brew installs on mac doesn't seem to allow mongo connector installation

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.

https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png
[cid:jira-generated-image-avatar-67f5a1cc-dd86-434e-805b-d81f6d06c17b] Robert Walters<https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.walters> commented on [Task] KAFKA-288<https://jira.mongodb.org/browse/KAFKA-288>

Re: Using brew installs on mac doesn't seem to allow mongo connector installation<https://jira.mongodb.org/browse/KAFKA-288>

Comment by Robert Walters [ 10/Feb/22 ]

I think there is some confusion with regard to what it means to install a connector on Kafka Connect.

What I am suggesting is use docker to launch Kafka Connect and the rest of the kafka services.  This just helps with cleanup as nothing is left installed on your mac.

within Kafka Connect you can install as many connectors as you'd like MongoDB, MySQL and whatever other connectors you'd like to use.  It doesn't have anything to do with Docker at that point they are just JARs added to a folder on that Kafka Connect instance.

if it is easier I'd be happy to jump on a zoom and go over this, just lmk and Ill email you directly.

Comment by Alexander Boese [ 10/Feb/22 ]

But I am trying to install three different connectors. You're suggesting I install the docker image with the mongo connector that works....okay great. But I also need the other connectors working. So if the others have similar issues, do I need to have a docker instance per connector? I mean, I'm not faulting the docker approach, but if you're really going to have a docker image for connectors, why not make one image with all the connectors? Then I only have to turn things off, which relatively speaking is trivial. Or maybe one with the most popular ones? Or, if you really have this stuff figured out, a way to select the connectors I need to supply me with a conforming image.

I'd honestly much rather have the documentation. If the only way to get that is by going through the code, then great. But I see no documentation atm and no recommendations on how to best troubleshoot these besides "use the docker instance" which again forces me to cobble things, so still not ideal.

Thoughts?

Mr. Alex Boese
CACI Inc. Federal
O: (202) 436-9888 (skype-based)
H: (443) 825-5760
alexander.boese@caci.com
alexander.a.boese.ctr@mail.mil
________________________________
From: Robert Walters (Jira) <jira@mongodb.org>
Sent: Thursday, February 10, 2022 8:25 AM
To: Boese, Alexander - US <Alexander.Boese@caci.com>
Subject: [MongoDB-JIRA] (KAFKA-288) Using brew installs on mac doesn't seem to allow mongo connector installation

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.

https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png
[cid:jira-generated-image-avatar-67f5a1cc-dd86-434e-805b-d81f6d06c17b] Robert Walters<https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.walters> commented on [Task] KAFKA-288<https://jira.mongodb.org/browse/KAFKA-288>

Re: Using brew installs on mac doesn't seem to allow mongo connector installation<https://jira.mongodb.org/browse/KAFKA-288>

Comment by Robert Walters [ 10/Feb/22 ]

Once your kafka connect service is running you can install as many connectors as you'd like on that kafka connect instance.   You do not have to stand up multiple kafka connect containers.  Just one container that is running kafka connect, that's it.  I find Docker to be the best way to quickly stand up a Kafka environment.  It leaves nothing behind on my mac when I'm done.  There might be other options but it isn't anything I've needed to explore.

If you are not that familiar with Docker and Kafka, it might be easiest to install the Confluent Platform.  https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html  Run through the Docker install on this quickstart and you'll a web UI, the Control Center to use to add connectors as well.

 

 

 

 

 

 

Comment by Alexander Boese [ 10/Feb/22 ]

No I'm not looking for a quick test. I'm doing some investigational testing, and I don't need or want to spin up a new docker for every single connector I plan on using. Isn't there a better way to access default documentation than through the docker instance?

Mr. Alex Boese
CACI Inc. Federal
O: (202) 436-9888 (skype-based)
H: (443) 825-5760
alexander.boese@caci.com
alexander.a.boese.ctr@mail.mil
________________________________
From: Robert Walters (Jira) <jira@mongodb.org>
Sent: Wednesday, February 9, 2022 5:47 PM
To: Boese, Alexander - US <Alexander.Boese@caci.com>
Subject: [MongoDB-JIRA] Robert Walters mentioned you on KAFKA-288 (Jira)

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.

https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png
[cid:jira-generated-image-avatar-284db11b-6cf1-4f3e-9432-44b463f31243] Robert Walters<https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.walters> mentioned you on [Task] KAFKA-288<https://jira.mongodb.org/browse/KAFKA-288>

Re: Using brew installs on mac doesn't seem to allow mongo connector installation<https://jira.mongodb.org/browse/KAFKA-288>

Alexander Boese<https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.boese%40caci.com> if what you are seeking is to spin up a quick dev/test on a mac, use docker and follow the quickstart

https://docs.mongodb.com/kafka-connector/master/quick-start/

[Add Comment] <https://jira.mongodb.org/browse/KAFKA-288#add-comment> Add Comment<https://jira.mongodb.org/browse/KAFKA-288#add-comment>

This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to jira.mongodb.org<https://jira.mongodb.org> using your JIRA, MongoDB Cloud Manager, or MongoDB Atlas credentials.

________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

Comment by Robert Walters [ 09/Feb/22 ]

alexander.boese@caci.com if what you are seeking is to spin up a quick dev/test on a mac, use docker and follow the quickstart

https://docs.mongodb.com/kafka-connector/master/quick-start/

 

Generated at Thu Feb 08 09:06:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.