Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-102905

p99 latency of list and stats operations increases over time

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Here is an example of the p99.9 of streams_listStreamProcessors latency from Agent's perspective.

      On a few pods, the latency slowly creeps up. Ping also slows down on a few pods.

      https://victoria-metrics.corp.mongodb.com/select/0/vmui/#/?g0.range_input=6h&g0.end_input=2025-03-26T14%3A22%3A01&g0.relative_time=none&g0.tab=0&g0.tenantID=0&g0.expr=histogram_quantile%280.999%2Csum%28rate%28mongohouse_streams_querier_latency_bucket%7Bnamespace%3D%22streams-prod%22%2Cop_name%3D%22ListStreamProcessors%22%7D%5B5m%5D%29%29+by+%28le%29%29

      1. Is this happening on pods with a restarting/failing processor?
      2. Does the list() and stats() stats latency correlate with the ping() latency? ping() is not a streams command and it's not code we own. 

            Assignee:
            Unassigned Unassigned
            Reporter:
            matthew.normyle@mongodb.com Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              None
              None
              None
              None