[KAFKA-143] int64 datatype converted into nested $numberLong json Created: 11/Aug/20  Updated: 27/Oct/23  Resolved: 11/Aug/20

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: J vH Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

The source connector generates a (schemaless) payload based on a database collection.

All document data types work as one would expect, except Int64.

A MongoDb document field 'level' of data type Int32 is translated into 

"level": 3

That looks good. Now, when the data type of field 'level' is Int64, the output is 

"level":

{    "$numberLong": "3" }

The nested $numberLong element and the number represented as a string/varchar indicate a bug? I'd expect just "level": 3 to be the output again.

When sinking data into a collection, the default MongoDb data type for integers is Int64. Sourcing from the same collection results in this unexpected output.

Bug or expected though mysterious behavior?

 

 



 Comments   
Comment by Ross Lawley [ 11/Aug/20 ]

Hi jeffrey.vanhelden@thewarehouse.co.nz,

Thanks for the ticket. MongoDB supports more types than JSON does, as such to keep type fidelity there is a MongoDB Extended Json Specification. This is the output you are seeing, which ensures that bson types can be round tripped.

There is a change in 1.3.0, specifically KAFKA-99 which allows configuration of Json output via the connector.

Ross

Generated at Thu Feb 08 09:05:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.