[KAFKA-284] Example for sink connector does not work Created: 26/Jan/22  Updated: 04/May/22  Resolved: 14/Feb/22

Status: Closed
Project: Kafka Connector
Component/s: Test
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Gleb Abroskin Assignee: Valentin Kavalenka
Resolution: Duplicate Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates KAFKA-286 Mongo sink connector must tolerate th... Closed

 Description   

Summary

What is the problem or use case, what are we trying to achieve?

I'm trying to run the docker example from mongo-kafka repo and see if it's possible to process mongo change stream with spark (it might be an antipattern, i'm just curious, how all this is going to function together).

Everything launches successfully, but there is no records in the mongo that should be sinked from the kafka topic. The connector in UI is in degraded state, although I didn't change anything from the master. Checked the `connect` container logs there was an error I don't fully understand

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/kafka/connect/sink/ErrantRecordReporter
        at com.mongodb.kafka.connect.sink.MongoSinkTask.nopErrorReporter(MongoSinkTask.java:143)
        at com.mongodb.kafka.connect.sink.MongoSinkTask.createErrorReporter(MongoSinkTask.java:123)
        at com.mongodb.kafka.connect.sink.MongoSinkTask.start(MongoSinkTask.java:73)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:300)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:189)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/connect/sink/ErrantRecordReporter
        ... 12 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.sink.ErrantRecordReporter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more

MY NOT EDUCATED GUESS: Looks like there was an error when deserializing the message, but connector was not able to handle it because the class is not there.

I've tried to replace 1.7.0 (current master) release with the 1.6.1 release, the result was the same.

Motivation

Who is the affected end user?

I guess, everyone who is trying to use mongo-kafka, that's why I placed major priority

How does this affect the end user?

End users can not try out the system, they are blocked.

How likely is it that this problem or use case will occur?

100% of the time, since I'm running docker-compose from repo without changes.

If the problem does occur, what are the consequences and how severe are they?

The connector in the example does not work, so the example does not work either.

Is this issue urgent?

IDK

Is this ticket required by a downstream team?

IDK

Is this ticket only for tests?

No

Cast of Characters

Engineering Lead:
Document Author:
POCers:
Product Owner:
Program Manager:
Stakeholders:

Channels & Docs

Slack Channel

[Scope Document|some.url]

[Technical Design Document|some.url]



 Comments   
Comment by Gleb Abroskin [ 14/Feb/22 ]

Thanks for the fix! Issue can be closed, I can not reproduce anymore.

Comment by Valentin Kavalenka [ 07/Feb/22 ]

Hi gabroskin@fun.co,

Thank you for reporting the problem. It has been fixed in master (future 1.7) by KAFKA-286.

I've tried to replace 1.7.0 (current master) release with the 1.6.1 release, the result was the same.

I don't think 1.6.1 can produce the same result. The only place that causes loading of the ErrantRecordReporter class is the try-catch block here https://github.com/mongodb/mongo-kafka/blob/r1.6.1/src/main/java/com/mongodb/kafka/connect/sink/MongoSinkTask.java#L165-L182, and you may see that it catches NoClassDefFoundError. As far as I can see, the absence of ErrantRecordReporter must not be a problem in 1.6.1.

If you are sure that the problem is present in 1.6.1, could you please attach the exact error you see when using 1.6.1?

Comment by Gleb Abroskin [ 26/Jan/22 ]

Forgot to add steps to reproduce:

./run.sh 

from the docker/ directory in the repo. (README says to run from docs directory)

Generated at Thu Feb 08 09:06:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.