Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-48388

MongoDB processes get hung up when trying to acquire lock

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.2
    • Component/s: None
    • None
    • ALL

      We have a three node MongoDB replica set deployed in our Prod environment. The primary mongod process gets hung up after running for 12 hours or so. We are able to see too many threads (around 15,000) stuck in the same stack,

       

      #0 0x00007efd9c853c21 in do_futex_wait () from /lib64/libpthread.so.0
      #1 0x00007efd9c853ce7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
      #2 0x00007efd9c853d85 in sem_timedwait () from /lib64/libpthread.so.0
      #3 0x00005580f6d94c6c in mongo::TicketHolder::waitForTicketUntil(mongo::Date_t) ()
      #4 0x00005580f681aedc in mongo::LockerImpl<false>::_lockGlobalBegin(mongo::LockMode, mongo::Duration<std::ratio<1l, 1000l> >) ()
      #5 0x00005580f680a724 in mongo::Lock::GlobalLock::_enqueue(mongo::LockMode, unsigned int) ()
      #6 0x00005580f680a79e in mongo::Lock::GlobalLock::GlobalLock(mongo::OperationContext*, mongo::LockMode, unsigned int, mongo::Lock::GlobalLock::EnqueueOnly) ()
      #7 0x00005580f680a7e8 in mongo::Lock::GlobalLock::GlobalLock(mongo::OperationContext*, mongo::LockMode, unsigned int) ()

       

      Attaching the pstack and diagnostic metrics.

      Due to the sensitive nature of the db.logs they cannot be shared.  The db logs had statements that showed around 14484 connections were open.

       

      2020-05-22T13:38:49.633+0000 I NETWORK [listener] connection accepted from 172.28.96.186:43437 #18988 (14484 connections now open)

       

        1. diagnostic.data.tar.gz
          14.82 MB
        2. pstatck_file
          58.96 MB

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            raghu.9208@gmail.com Raghu c
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: