Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-247

Recreate change stream from the point of failure for event > 16 MB

    XMLWordPrintableJSON

Details

    • Icon: Improvement Improvement
    • Resolution: Won't Fix
    • Icon: Unknown Unknown
    • None
    • None
    • Source

    Description

      When a change event in the change stream exceeds the 16MB limit, existing change stream is closed with an exception and new change stream is opened. In a system with a higher update load this will likely miss the change events in the time it takes to start a new change stream. I have 2 proposal for improvement.

      Solution #1

      The error message of the exception contains the resumeToken of the failed event. Use the "ChangeStream.startAfter(<resumeToken>)" to start the new stream just after the failed event, leading to zero loss of events.

      Example error message

      BSONObj size: 19001449 (0x121F069) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: "826115AEE9000000012B022C0100296E5A1004D148D6B22E8F49B3A65DAE80A4683566463C5F6964003C316663726B36326F6D30303030303030000004" }
      

      Solution #2

      Increment the "clusterTime" (introduced in v4.0) available in the MongoCommandException, by 1 ordinal and use it with "ChangeStream.startAtOperationTime(<BsonTimestamp>)"

      For sharded cluster, it is possible that multiple events may have same cluster time and this approach can skip few good events with same timestamp as the bad one.

      Attachments

        Activity

          People

            Unassigned Unassigned
            dhruvangmakadia1@gmail.com Dhruvang Makadia
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: