Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-95499

Azure smoke test processor stopped processing messages

XMLWordPrintableJSON

    • Atlas Streams
    • Fully Compatible
    • Sprint 59, Sprint 60

      See this investigation doc: https://docs.google.com/document/d/1g5d3zq3m4VGGg0F5a7DRLK_oWdf4oOduBgHsHkMjGRM/edit 

      ====

      This also happened around 10/3/2024:

      Stack trace record:

      https://splunk.corp.mongodb.com/en-US/app/cloud/search?earliest=1727974317&latest=1727975217&q=search%20index%3Dmhouse-staging%20streams-spp-canary-86bdd58f58-9v5zm%20msg!%3D%22Connection%20accepted%22%20msg!%3D%22WiredTiger%20message%22%20msg!%3D%22Connection%20ended%22%20c!%3DSTREAMS&display.page.search.mode=smart&dispatch.sample_ratio=1&display.page.search.tab=events&display.general.type=events&sid=1727975331.4432991

       

       

       

      https://splunk.corp.mongodb.com/en-US/app/cloud/search?earliest=1727798400&latest=1727971896&q=search%20index%3Dmhouse-staging%20(kanopy_immortal_smoke_test_8fc9ad56_08997cbe%20OR%2066fac1acb5cdfac68231529a)%20msg!%3D%22Stream%20processor%20state%22%20c%3DSTREAMS&display.page.search.mode=smart&dispatch.sample_ratio=1&display.page.search.tab=events&display.general.type=events&sid=1727971918.4428255

       

      On azure staging, pod host=streams-spp-canary-86bdd58f58-9v5zm

       

      mongostream thinks it’s running kanopy_immortal_smoke_test_8fc9ad56_08997cbe, but the processor is not processing input messages.

      it’s not fully stuck, the runOnce metric is incrementing. but it’s not picking up any new messages. I even tried manually writing to the collection that sources it.
      https://victoria-metrics.corp.mongodb.com/select/0/vmui/#/?g0.expr=mongohouse_mstream_runonce_count%7Bprocessor_id%3D%2266fac1acb5cdfac68231529a%22%7D&g0.range_input=3h&g0.end_input=2024-10-03T16%3A46%3A49&g0.tab=0&g0.relative_time=none&g0.tenantID=0

            Assignee:
            mayuresh.kulkarni@mongodb.com Mayuresh Kulkarni
            Reporter:
            matthew.normyle@mongodb.com Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: