Per input/output latency measurements for change stream $source and kafka sink

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      AA wants latency measurements for each input collection / output topic pair. This ticket covers adding the engine stats for this.

      This will build on this PR. Below is a rough explanation of what to do:

      1. Put these code changes under a feature flag (set to false for now).
      2. Confirm the new stats() response with Joe and Laura. Let's assume a schema like below for now:
      stats() response
      
      {
        ...
        latency: {
          ... existing fields p50,p99,sum,count,start,end ...,
          targets: [
             {
               input: {connectionName, db, coll},
               output: {connectionName, topic},
               p50, p99, sum, count
             },
             ...
          ]
        }
      }

      2. Add unordered_map<Target, Summary> perInputOutputLatency in OperatorStats... this is only set by Sink operator (just like overallLatency).

      3. Amend ChangeStreamSourceOperator to seta new Target StreamDocument::inputTarget field in StreamDocument.

      4. Amend KafkaEmitOperator to add measurements to perInputOutputLatency. 

            Assignee:
            Jada Lilleboe
            Reporter:
            Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: