[KAFKA-232] Add date field conversion post processor Created: 02/Jul/21  Updated: 28/Oct/23  Resolved: 12/Jul/21

Status: Closed
Project: Kafka Connector
Component/s: Sink
Affects Version/s: None
Fix Version/s: 1.6.0

Type: Improvement Priority: Unknown
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to KAFKA-228 Support configuration / creation of n... Closed
Documentation Changes: Needed
Documentation Changes Summary:
Config Description Field type Default Value Validation
timeseries.timefield.auto.convert Converts the timeseries field to a BSON datetime type. If the value is a number value it will use it as the milliseconds from epoch. If the value is a String it will use `timeseries.timefield.auto.convert.date.format` configuration to parse the date. boolean false
timeseries.timefield.auto.convert.date.format The DateTimeFormatter pattern to use when converting String dates.Defaults to supporting UTC ISO date times. string yyyy-MM-dd['T'][ ]HH:mm:ss[.][SSSSSS][SSS]['Z'] A valid DateTimeFormatter format

 Description   

The conversion SMT has some limitations, so for ease of use lets add a post process that handles json dates and longs / ints and converts them to BsonDateTime.

Created an automatic post processor that will process documents when timeseries.timefield.auto.convert=true.
For Strings yyyy-MM-dd['T'][ ]HH:mm:ss[.][SSSSSS][SSS]['Z'] is the default format used. So supports all of the following formatted strings: "2021-07-13T12:00:00.000001Z", "2021-07-13 12:00:00.000001", "2021-07-13T12:00:00.001Z", "2021-07-13 12:00:00.001", "2021-07-13T12:00:00Z", "2021-07-13 12:00:00"

Failing to convert the field does not error in the processor and the timefield is left as is.



 Comments   
Comment by Githook User [ 12/Jul/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Improved timeseries usability

Added conversion tool for non date (String, longs) time data

KAFKA-232
Branch: master
https://github.com/mongodb/mongo-kafka/commit/d352b5a3a789f646d8adc6871306011ff39896fa

Comment by Ross Lawley [ 06/Jul/21 ]

PR: https://github.com/mongodb/mongo-kafka/pull/77

Generated at Thu Feb 08 09:05:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.