[KAFKA-278] Invalid value in config of output.schema.key Created: 18/Jan/22  Updated: 27/Oct/23  Resolved: 19/Jan/22

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: jiazhi zhou Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: external-user, web-serivce
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2022-01-18-17-26-07-484.png    

 Description   

I got an error when I execute this command, can anyone help me?

curl -X POST \
--url http://localhost:8083/connectors \
-H "Content-Type: application/json" \
--data '
{
 "name": "zbcde",
 "config": {
 "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
 "connection.uri": "mongodb://localhost:27017/?authSource=admin",
 "database": "test",
 "collection": "test_kafka",
 "copy.existing" :"true",
 "publish.full.document.only":"true",
 "change.stream.full.document":"updateLookup",
 "output.format.key":"schema",
 "output.schema.key":"{\"name\":\"keySchema\",\"type\":\"record\",\"fields\":[{\"name\": \"fullDocument\",\"type\": \"record\",\"fields\":[ { \"name\": \"a\", \"type\": \"string\" },{ \"name\": \"b\", \"type\": \"string\" },{ \"name\": \"c\", \"type\": \"string\" }]}]}",
 "output.json.formatter":"com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson"
 }
}'



 Comments   
Comment by Ross Lawley [ 19/Jan/22 ]

Hi zhoujiazhiwork@163.com,

Thank you for reaching out. The Avro schema parser doesn't accept the avro schema definition. The cause is because the fullDocument type cannot defined as a record inline using that syntax, it has to be nested. Its a pain because its shorter than the official way, which is to nest the record definition in its own object:

    String jsonSchema = ""
            + "{\"name\":\"keySchema\", \"type\":\"record\", \"fields\": ["
            + "   {\"name\": \"fullDocument\", \"type\": {"
            + "     \"type\": \"record\", \"name\": \"fullDocumentRecord\","
            + "     \"fields\": ["
            + "        {\"name\": \"a\", \"type\": \"string\"},"
            + "        {\"name\": \"b\", \"type\": \"string\"},"
            + "        {\"name\": \"c\", \"type\": \"string\"}]"
            + "     }"
            + " }"
            + "]}";

Personally, I'd prefer a shorter method but the official org.apache.avro.Schema.Parser won't accept it.

For future reference I wanted to give you some resources to get this question answered more quickly in the future.

  • Our MongoDB support portal, located at support.mongodb.com
  • Our MongoDB community portal, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI

All the best,

Ross Lawley

Generated at Thu Feb 08 09:06:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.