Streams: Expose kafka per-partition watermark in verbose stats()

XMLWordPrintableJSON

    • Atlas Streams
    • Fully Compatible
    • Sprint 56
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      We currently only expose the combined watermark in the kafka source in the stats(). We want to expose per-partition watermark information in verbose stats as well.

      In the KafkaConsumerOperator, there is a watermark for each input partition.

      We can expose the partition watermark as a "watermark" field within the KafkaConsumerPartitionState struct:

      https://github.com/10gen/mongo/blob/master/src/mongo/db/modules/enterprise/src/streams/commands/stream_ops.idl#L134

      We can test this new functionality in the below places:

      • kafka_consumer_operator_test (run with ninja +kafka_consumer_operator_test -j400). This tests using mock Kafka classes.
      • kafka.js test (run with s-mongo-build && test-streams-kafka). This tests against a real kafka broker running in a local docker container.

       

              Assignee:
              Calvin Nix
              Reporter:
              Aadesh Patel (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 1 week
                  1w
                  Remaining:
                  Remaining Estimate - 1 week
                  1w
                  Logged:
                  Time Spent - Not Specified
                  Not Specified