Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-83885

Fix hang that can occur in start due to a failing Kafka connection

    XMLWordPrintableJSON

Details

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Major - P3 Major - P3
    • 7.3.0-rc0
    • None
    • None
    • None
    • Atlas Streams
    • Fully Compatible
    • Sprint 37

    Description

      mongostreams stop request is timing out after a minute.
      (splunk)looks like it has run into some deadlock. Taking a process dump and then will try to mitigate it.
      kb-copy-mongoclient streams-spp-6bc9946bc-8tdkn
      kb-bash-mstreams streams-spp-6bc9946bc-8tdkn
      MongoDB Enterprise > db.runCommand({"streams_listStreamProcessors": ""})
      ... this hangs ...

      matthew.normyle  7 minutes ago

      Got the stack traces with
      kb-copygdb streams-spp-6bc9946bc-8tdkn
      kb-bash-mstreams streams-spp-6bc9946bc-8tdkn
      /tmp/gdb attach 1
       
      #0 0x00007f25dcf7174a in pthread_join () from /lib64/libpthread.so.0
      #1 0x0000558f2c7ef404 in thrd_join ()
      #2 0x0000558f2c6d4b1c in rd_kafka_destroy_app ()
      #3 0x0000558f2c6afe3f in RdKafka::ConsumerImpl::~ConsumerImpl() ()
      #4 0x0000558f2c62ac27 in streams::KafkaPartitionConsumer::~KafkaPartitionConsumer() ()
      #5 0x0000558f2c62adf2 in streams::KafkaPartitionConsumer::~KafkaPartitionConsumer() ()
      #6 0x0000558f2c61a3e8 in streams::KafkaConsumerOperator::doStop() ()
      #7 0x0000558f2c65cbf1 in streams::OperatorDag::stop() ()
      #8 0x0000558f2c604b2a in streams::Executor::stop() ()
      #9 0x0000558f2c5e348c in streams::StreamManager::startStreamProcessor(mongo::StartStreamProcessorCommand const&) ()
       
      #2 0x0000558f30e1df70 in mongo::latch_detail::Mutex::lock() ()
      #3 0x0000558f2c5e285d in streams::StreamManager::stopStreamProcessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
      #4 0x0000558f2c5d788c in mongo::TypedCommand<streams::StopStreamProcessorCmd>::InvocationBase::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*) ()

      Attachments

        Activity

          People

            matthew.normyle@mongodb.com Matthew Normyle
            matthew.normyle@mongodb.com Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: